Image processing apparatus, image processing method, computer program and computer readable recording medium

ABSTRACT

Disclosed is an image processing method. The method includes the steps of receiving an image obtained from a plurality of vehicles positioned on a road, storing the received images according to acquisition information of the received images; determining a reference image and a target image based on images having the same acquisition information among the stored images, performing an image registration using a plurality of feature points extracted from each of the determined reference image and target image, performing a transparency process for each of the reference image and the target image performed with the image registration, extracting static objects from the transparency-processed image, and comparing the extracted static objects with objects on an electronic map pre-stored to updating the electronic map data, when the objects on the electronic map data pre-stored are different from the extracted static objects.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part application of U.S.application Ser. No. 17/308,105 filed on May 5, 2021, which is acontinuation application of U.S. application Ser. No. 17/011,610 filedon Sep. 3, 2020, which is a continuation application of U.S. applicationSer. No. 16/226,759 filed on Dec. 20, 2018 which is a continuationapplication of U.S. application Ser. No. 15/822,705 filed on Nov. 27,2017, which claims priority to Korean Patent Application Nos.10-2016-0158832 filed on Nov. 26, 2016, and 10-2017-0148115 filed onNov. 8, 2017, with the Korean Intellectual Property Office, the entiredisclosures of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, a computer program, and a computer readable recordingmedium, and more particularly, to an image processing apparatus, animage processing method, a computer program, and a computer readablerecording medium for providing a map having high accuracy using imagesobtained through cameras.

2. Description of the Related Art

Although automakers are recently trying to implement an autonomousvehicle with a higher level of advanced driver assistance system (ADAS),they are facing limitations due to the technical problems ofconventional systems using sensors such as cameras and radar. In orderto overcome such a limitation, the automakers try to find a solutionutilizing additional information system to implement the autonomousvehicle, and an example of the representative additional informationsystem is a detailed map.

The detailed map refers to a map including information on positions andtypes of all fixed objects on the road, and the detailed map serves tocomplement performance of the sensor in a situation in which it isdifficult for the sensor to normally operate. In a case in whichinformation of the sensor is not correct due to obstacles or badweather, the vehicle may complement the incorrect information byutilizing information of the detailed map.

Since the detailed map should provide information on the positions andtypes of all fixed geographic features on the road, it is important toquickly reflect accuracy of the map and changes of an actual road andthe geographic features. However, when it is difficult to know theaccurate information on the types of fixed geographic features due tothe geographic features covered by a moving object, or changes in a roadenvironment such as an installation of a traffic light,extending/closing of the road, a change in lane information, and thelike occur, there is a problem that such information is not quicklyreflected on the actual map. Therefore, a method capable of solving sucha problem is required.

SUMMARY

An aspect of the present invention may provide an image processingapparatus, an image processing method, a computer program, and acomputer readable recording medium for creating a map.

An aspect of the present invention may also provide an image processingapparatus, an image processing method, a computer program, and acomputer readable recording medium for creating map data using imagesobtained by a camera mounted in a vehicle.

An aspect of the present invention may also provide an image processingapparatus, an image processing method, a computer program, and acomputer readable recording medium for updating map data using imagestransmitted from vehicles positioned on a road.

According to an aspect of the present invention, an image processingmethod may include receiving images obtained from a plurality ofvehicles positioned on a road; storing the received images according toacquisition information of the received images; determining a referenceimage and a target image based on images having the same acquisitioninformation among the stored images; performing an image registrationusing a plurality of feature points extracted from each of thedetermined reference image and target image; performing a transparencyprocess for each of the reference image and the target image which areimage-registered; extracting static objects from thetransparency-processed image; and comparing the extracted static objectswith objects on map data which is previously stored and updating the mapdata when the objects on the map data which is previously stored and theextracted static objects are different from each other.

The performing of the image registration may include extracting theplurality of feature points from each of the determined reference imageand target image; and performing the image registration for thedetermined images using the plurality of extracted feature points.

The plurality of feature points may be points at which image brightnessvalue suddenly changes in the reference image or the target image andmay be edges of pixels or corner points.

The transparency process may multiply R, G, and B pixel values ofrespective pixels included in the images for which the transparencyprocess is to be performed by a predetermined value smaller than 1, andthe predetermined value may be a reciprocal number of N, which is atotal number of the images for which the transparency process is to beperformed.

The acquisition information may include at least one of information onpositions at which the images are photographed, information on angles atwhich the images are photographed, and information on directions inwhich the images are photographed.

The extracting of the plurality of feature points may include extractinga plurality of first feature points from the reference image; andextracting a plurality of second feature points from the target image.

The performing of the image registration may include performing amatching operation matching a first feature point group in which theplurality of first feature points are grouped and a second feature pointgroup in which the plurality of second feature points are grouped;calculating a homography using information of matched pairs between thefirst feature point group and the second feature point group through thematching operation; converting the target image using the calculatedhomography; and registering the reference image and the converted targetimage.

The updating of the map data may include confirming position informationof the extracted static objects; examining whether or not objectsdifferent from the extracted static objects exist at a positioncorresponding to the confirmed position information of the staticobjects in the map data which is previously stored; and updating the mapdata by reflecting the extracted static objects to the map data which ispreviously stored corresponding to the position information, when thedifferent objects exist as a result of the examination.

The image processing method may further include transmitting the updatedmap data to the plurality of vehicles positioned on the road.

According to another aspect of the present invention, an imageprocessing apparatus may include a receiving unit receiving imagesobtained from a plurality of vehicles positioned on a road; a storingunit storing the received images according to acquisition information ofthe received images; a controlling unit determining a reference imageand a target image based on images having the same acquisitioninformation among the stored images; and an image processing unitperforming an image registration using a plurality of feature pointsextracted from each of the determined reference image and target imageand performing a transparency process for each of the reference imageand the target image which are image-registered, wherein the controllingunit extracts static objects from the transparency-processed image,compares the extracted static objects with objects on map data which ispreviously stored, and updates the map data when the objects on the mapdata which is previously stored and the extracted static objects aredifferent from each other.

The image processing unit may generate one synthesized image byextracting the plurality of feature points from each of the determinedreference image and target image and performing the image registrationfor the determined images using the plurality of extracted featurepoints.

The plurality of feature points may be points at which image brightnessvalue suddenly changes in the reference image or the target image and beedges of pixels or corner points.

The transparency process may multiply R, G, and B pixel values ofrespective pixels included in the images for which the transparencyprocess is to be performed by a predetermined value smaller than 1, andthe predetermined value may be a reciprocal number of N, which is atotal number of the images for which the transparency process is to beperformed.

The acquisition information may include at least one of information onpositions at which the images are photographed, information on angles atwhich the images are photographed, and information on directions inwhich the images are photographed.

The image processing unit may include a feature point extracting unitextracting a plurality of first feature points from the reference imageand extracting a plurality of second feature points from the targetimage; a feature point matching unit performing a matching operationmatching a first feature point group in which the plurality of firstfeature points are grouped and a second feature point group in which theplurality of second feature points are grouped; a homography calculatingunit calculating a homography using information of matched pairs betweenthe first feature point group and the second feature point group throughthe matching operation; an image registration unit converting the targetimage using the calculated homography and registering the referenceimage and the converted target image; and a transparency processing unitperforming a transparency process for the registered images.

The controlling unit may confirm position information of the extractedstatic objects, examines whether or not objects different from theextracted static objects exist at a position corresponding to theconfirmed position information of the static objects in the map datawhich is previously stored, and update the map data by reflecting theextracted static objects to the map data which is previously storedcorresponding to the position information, when the different objectsexists as a result of the examination.

According to another aspect of the present invention, an imageprocessing method may include selecting images for the same region amonga plurality of images photographed by a moving body; performing atransparency process for each of the selected images; registering thetransparency-processed images; and determining static objects in theregistered image based on transparencies of objects included in theregistered image.

In the selecting of the images, images having the same acquisitioninformation for each of the plurality of images may be selected, and theacquisition information may include at least one of information onpositions at which the images are photographed, information on angles atwhich the images are photographed, and information on directions inwhich the images are photographed.

The determining of the static objects may include calculating standarddeviations of a pixel value of the registered image, a pixel value ofthe reference image, and pixel values of target images for each ofpixels; and determining pixels of which the calculated standarddeviation is a predetermined value or less as pixels for the staticobjects and determining pixels of which the calculated standarddeviation exceeds the predetermined value as pixels for the dynamicobjects.

The image processing method may further include excluding the dynamicobjects from the registered image, confirming position information ofthe static objects in the registered image from which the dynamicobjects are excluded, and updating map data based on the positioninformation of the static objects.

According to another exemplary embodiment of the present invention, acomputer readable recording medium in which a program for executing animage processing method is recorded may be provided.

According to another exemplary embodiment of the present invention, acomputer program stored in a computer readable recording medium toexecute an image processing method may be provided.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a map creating system according to anexemplary embodiment of the present invention;

FIG. 2 is a diagram illustrating a block configuration of an electronicapparatus mounted in first to third vehicles according to an exemplaryembodiment of the present invention;

FIG. 3 is a diagram illustrating a block configuration of an imageprocessing apparatus according to an exemplary embodiment of the presentinvention;

FIG. 4 is a diagram illustrating operations of a feature pointextracting unit and a feature point matching unit of an image processingunit according to an exemplary embodiment of the present invention;

FIG. 5 is a diagram illustrating a process of matching a reference imagewith a target image according to a feature point matching described inFIG. 4 according to an exemplary embodiment of the present invention;

FIG. 6 is a diagram illustrating a matching relationship between thereference image and the target image by a homography calculated by ahomography calculating unit of the image processing unit according to anexemplary embodiment of the present invention;

FIG. 7 is a diagram illustrating a distinction between a static objectand a dynamic object according to an exemplary embodiment of the presentinvention;

FIGS. 8 to 11 are diagrams illustrating a transparency process accordingto an exemplary embodiment of the present invention;

FIG. 12 is a diagram illustrating image processing results for thereference image and the target image according to an exemplaryembodiment of the present invention;

FIG. 13 is a diagram illustrating an operation flow of an imageprocessing apparatus according to an exemplary embodiment of the presentinvention;

FIG. 14 is a diagram illustrating an operation (S1325) of FIG. 13 indetail;

FIG. 15 is a diagram illustrating a block configuration of an imageprocessing apparatus for creating a map according to another exemplaryembodiment of the present invention;

FIG. 16 is a diagram illustrating a method for creating a map of theimage processing apparatus of FIG. 15;

FIG. 17 is a diagram illustrating a block configuration of an electronicapparatus according to still another exemplary embodiment of the presentinvention;

FIG. 18 is a diagram illustrating an operation flow of the electronicapparatus of FIG. 17;

FIGS. 19 and 20 are diagrams illustrating a transparency process forfive images;

FIG. 21 is a diagram illustrating an image registration of thetransparency-processed images of FIG. 20;

FIG. 22 is a diagram illustrating a result of the image registrationaccording to FIG. 21;

FIG. 23 shows a simplified block diagram of an electronic deviceaccording to various embodiments;

FIG. 24 shows a simplified block diagram of a server according tovarious embodiments;

FIGS. 25 and 26 show diagrams for explaining an embodiment foridentifying a geographic location of a fixed external object accordingto various embodiments;

FIGS. 27 and 28 show schematic diagrams for explaining an embodiment foridentifying a geographic location of a fixed external object accordingto various embodiments;

FIG. 29 shows a schematic view for explaining an embodiment foracquiring information on a fixed external object according to variousembodiments;

FIGS. 30 to 32 each show examples of operations of an electronic deviceaccording to various embodiments;

FIGS. 33 and 34 show examples of operations of a server according tovarious embodiments;

FIG. 35 shows an example for identifying a geographic location of afixed external object according to various embodiments;

FIG. 36 shows an example for identifying a geographic location of afixed external object according to various embodiments;

FIG. 37 shows an example of an operation of an electronic deviceaccording to various embodiments;

FIG. 38 shows another example of an operation of an electronic deviceaccording to various embodiments;

FIG. 39 shows another example of an operation of an electronic deviceaccording to various embodiments; and

FIG. 40 shows an example of an operation of a first electronic deviceaccording to various embodiments.

DETAILED DESCRIPTION

The following description illustrates only a principle of the presentinvention. Therefore, those skilled in the art may implement theprinciple of the present invention and devise various apparatusesincluded in the spirit and scope of the present invention although notclearly described or shown in the present specification. In addition, itis to be understood that all conditional terms and exemplary embodimentsmentioned in the present specification are obviously intended only toallow those skilled in the art to understand a concept of the presentinvention in principle, and the present invention is not limited toexemplary embodiments and states particularly mentioned as such.

Further, it is to be understood that all detailed descriptionsmentioning specific exemplary embodiments of the present invention aswell as principles, aspects, and exemplary embodiments of the presentinvention are intended to include structural and functional equivalencesthereof. Further, it is to be understood that these equivalences includean equivalence that will be developed in the future as well as anequivalence that is currently well-known, that is, all devices devisedso as to perform the same function regardless of a structure.

Therefore it is to be understood that, for example, a block diagram ofthe present specification shows a conceptual aspect of an illustrativecircuit for embodying a principle of the present invention. Similarly,it is to be understood that all flowcharts, state transition views,pseudo-codes, and the like show various processes that may tangiblyembodied in a computer-readable medium and that are executed bycomputers or processors regardless of whether or the computers or theprocessors are clearly illustrated.

Functions of various devices including processors or functional blocksrepresented as concepts similar to the processors and illustrated in theaccompanying drawings may be provided by hardware having capability toexecute appropriate software as well as dedicated hardware. When thefunctions are provided by the processors, the above-mentioned functionsmay be provided by a single dedicated processor, a single sharedprocessor, or a plurality of individual processors, in which some ofthem may be shared.

In addition, terms mentioned as a processor, a control, or a conceptsimilar to the processor or the control should not be interpreted toexclusively cite hardware having capability to execute software, butshould be interpreted to implicitly include digital signal processor(DSP) hardware and a read only memory (ROM), a random access memory(RAM), and a non-volatile memory for storing software without beinglimited thereto. The above-mentioned terms may also include well-knownother hardware.

In the claims of the present specification, components represented asmeans for performing functions mentioned in a detailed description areintended to include all methods for performing functions including alltypes of software including, for example, a combination of circuitdevices performing these functions, firmware/micro codes, or the like,and are coupled to appropriate circuits for executing the software. Itis to be understood that since functions provided by variously mentionedmeans are combined with each other and are combined with a schemedemanded by the claims in the inventions defined by the claims, anymeans capable of providing these functions are equivalent to meansrecognized from the present specification.

The above-mentioned objects, features, and advantages will becomeobvious from the following detailed description provided in relation tothe accompanying drawings. Therefore, those skilled in the art to whichthe present invention pertains may easily practice a technical idea ofthe present invention. Further, in describing the present invention, inthe case in which it is judged that a detailed description of awell-known technology associated with the present invention mayunnecessarily make unclear the gist of the present invention, it will beomitted.

Hereinafter, various exemplary embodiments of the present invention willbe described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram illustrating a map creating system 10 according toan exemplary embodiment of the present invention.

Referring to FIG. 1, a map creating system 10 according to the presentinvention includes first to third vehicles 100, 110, and 120, and animage processing apparatus 130.

The first to third vehicles 100, 110, and 120 are mounted with cameras,which are apparatuses capable of obtaining images, and transmit theobtained images to the image processing apparatus 130 while beingpositioned on a road. Although FIG. 1 illustrates the vehicles, othermeans other than the vehicle, for example, a movable means such as aperson, a bicycle, a ship, a train, or the like, may also be implementedas long as they may transmit the images obtained by photographing thesubjects on the road or around the road to the image processingapparatus 130. Such a movable means will be collectively referred to asa moving body. Hereinafter, a case in which the moving body is thevehicle will be described as an example, for convenience of explanation.

Meanwhile, the subjects described above include fixed static objectsthat may be reflected in map data such as bridges, buildings, roads,sidewalks, road construction marks, speed bumps, crosswalks,intersections, traffic lights, median strips, bus stops, directionalsigns, and the like.

Further, the image processing apparatus 130 according to an exemplaryembodiment of the present invention may process images received from thefirst to third vehicles 100, 110, and 120 to create an electronic map,and may transmit the created electronic map to other vehicles positionedon the road including the first to third vehicles 100, 110, and 120.

Further, the image processing apparatus 130 according to an exemplaryembodiment of the present invention may also be a map creating server.

FIG. 2 is a diagram illustrating a block configuration of an electronicapparatus 200 mounted in the first to third vehicles 100, 110, and 120according to an exemplary embodiment of the present invention.

Referring to FIG. 2, an electronic apparatus 200 according to thepresent invention includes an input unit 210, a communicating unit 230,a camera unit 240, and a controlling unit 250, and may further include adisplay unit 220. As an example, the electronic apparatus 200, which isan apparatus capable of photographing an image, may be a mobile terminalsuch as a smartphone including a vehicle video camera or a camera.Further, the electronic apparatus 200 may be embedded in the vehicle soas to be connected to an electronic controller unit (ECU) of the vehiclethrough controller area network (CAN) communication, and may also be amobile terminal which may be held in the vehicle such as the smartphoneof the user and may transmit and receive data by being connected to amobile communication system.

The inputting unit 210 may receive a command or the like for performingan operation from the user and may include a key-pad, a dome switch, atouch pad (resistive/capacitive), a jog wheel, a jog switch, a pushswitch, or the like.

The communicating unit 230 may transmit the image photographed by thecamera unit 240 to the image processing apparatus 300, or transmit thephotographed image to other communication means included in the vehicle.The communicating unit 230 may perform communication in a wiredcommunication scheme as well as various wireless communication schemessuch as Bluetooth, Wi-Fi, wireless broadband, 3rd generation (3G), WCDMAscheme, long term evolution (LTE), and a 4th generation (4G)communication schemes.

The camera unit 240 converts an image signal (e.g., light) received fromthe outside into a specific electric signal to generate image data. Forexample, the camera unit 240 may obtain image data of the inside andoutside of the vehicle related to the driving of the vehicle.

The display unit 220 may output visual information to the user, and mayoutput a map or an image photographed during the driving. The displayunit 220 and the input unit 210 may be integrally formed as a touchpanel or a touch screen.

The controlling unit 250 generally controls the electronic apparatus200, for example, the controlling unit controls the camera unit 240 soas to photograph the image, and transmits the photographed image to theimage processing apparatus 130 through the communicating unit 230. Inaddition, since the controlling unit 250 also transmits acquisitioninformation including at least one of photographed position informationof the image, photographed angle information thereof, and photographeddirection information thereof when transmitting the image photographedby the camera unit 240 to the image processing unit 300 through thecommunicating unit 230, the image processing apparatus 300 according tothe present invention may classify the received images according to theacquisition information, thereby making it possible to quickly updatethe map data only for a point at which the static object is changed.Further, the controlling unit 250 may perform a camera calibrationprocess for adjusting camera parameters for the image photographed bythe camera unit 240 and transmit it to the image processing apparatus300 through the communicating unit 230.

Meanwhile, the electronic apparatus 200 mounted in the first to thirdvehicles 100, 110, and 120 may photograph the object in units ofpredetermined time, and transmit images of the photographed object tothe image processing apparatus 300. As an example, the electronicapparatus 200 may photograph the object in unit of one minute andtransmit images of the photographed object to the image processingapparatus 300. Since the images of the photographed object includeinformation on a photographed time and place, the photographed imagesmay be classified by the same place.

Further, according to an exemplary embodiment of the present invention,the controlling unit 250 of the electronic apparatus 200 may correctparameters such as positions, angles, features, and the like of thecameras installed in the respective vehicles through camera calibration.The reason for performing such a camera calibration process is that thephotographed place and target of the images photographed by the camerasare the same as each other, but installation heights, angles, and thelike of the cameras photographing the respective images are differentfrom each other, and therefore, an image registration may be easilyperformed only by correcting such characteristic parameters of thecamera.

Hereinafter, it is assumed that since the electronic apparatus 200 ismounted in the first to third vehicles 100, 110, and 120, the first tothird vehicles 100, 110, and 120 photograph the images and transmit thephotographed images to the image processing apparatus 130.

FIG. 3 is a block configuration of an image processing apparatus 300according to an exemplary embodiment of the present invention. Referringto FIG. 3, the image processing apparatus 300 according to the presentinvention includes all or some of a controlling unit 305, a receivingunit 310, a transmitting unit 315, a storing unit 320, and an imageprocessing unit 330.

The receiving unit 310 receives images obtained by the cameras mountedin a plurality of vehicles positioned on the road and acquisitioninformation including at least one of information on positions at whichthe obtained images are photographed, information on angles at which theobtained images are photographed, and information on direction in whichthe obtained images are photographed through wireless communication suchas long term evolution (LTE) or Wi-Fi, or wired communication.

The reason that the receiving unit 310 receives the obtained informationother than the images is to allow the controlling unit 305 to easilyclassify the obtained images for image processing. Specifically, sincethe images obtained at the same point may have different photographingdirections and angles, it is preferable that the controlling unit 305classifies the images in consideration of the different photographingdirection and angles and then performs the image processing according toan exemplary embodiment of the present invention.

For example, when the vehicle enters an intersection, the imageprocessing apparatus 300 according to the exemplary embodiment of thepresent invention should be able to confirm whether the obtained imageis an image obtained while the vehicle enters the intersection in eitherdirection, so that the target images may be accurately selected.

That is, in the case of the intersection of four directions, since thevehicle may obtain the images while entering in the four directions,respectively, it is possible to accurately create or update map dataaround the intersection only in a case in which the image processingapparatus 300 may distinguish the photographed directions of therespective images.

Further, in a case in which the image processing apparatus 300 accordingto the exemplary embodiment of the present invention considers theinformation on the photographed angles of the obtained images, it ispossible to create or update the map data through a more accurate imageprocessing.

The transmitting unit 315 transmits the map data updated or the map datastored in the storing unit 320 by the controlling unit 305 to theelectronic apparatus 200 of the vehicle of the user or the mobileterminal of the user by a control of the controlling unit 305.

The controlling unit 305 classifies the map data and the images receivedthrough the receiving unit 310 according to the obtained information ofthe received images and stores them in the storing unit 320. The mapdata is stored in a map data storing unit 320 a as map data which iscreated previously, and the received images are each stored in areceived image storing unit 320 b. The controlling unit 305 may performthe image processing for images having the same position information,direction information, or the like using the classified and storedimages, according to the exemplary embodiment of the present invention.Here, the storing unit 320 may be implemented as an embedded type ofstorage element such as a random access memory (RAM), a flash memory, aread only memory (ROM), an erasable programmable ROM (EPROM), anelectronically erasable and programmable ROM (EEPROM), a register, ahard disk, a removable disk, a memory card, or the like, as well as aremovable type of storage element such as a USB memory, or the like.

Such a storing unit 320 may be implemented within the image processingapparatus 300, and may be implemented in a type of external database(DB) server connected to the image processing apparatus 300.

Meanwhile, the controlling unit 305 determines a reference image and atarget image based on images having the same obtained information amongthe images stored in the received image storing unit 320 b, performs animage registration using a plurality of feature points extracted fromthe determined reference image and target image, respectively, andcontrols the image processing unit 330 so that a transparency process isperformed for the image registered images. Here, the reference imagerefers to an image that is a reference for the image registration, andthe target image refers to an image that is a target for the imageregistration.

Further, the controlling unit 305 registers the imagestransparency-processed by the image processing unit 330, and controlsthe image processing unit 330 to detect a static object region from theregistered image. In this case, the image processing unit 330 may eachcalculate standard deviations of a pixel value of the registered image,a pixel value of the obtained reference image, and pixel values of thetarget images converted according to a homography for each of thepixels, and may detect a static object region from the registered imagebased on the calculated standard deviation.

In addition, the controlling unit 305 compares detected static objectswith objects on the map data which is previously stored, and updates themap data stored in the map data storing unit 320 a, when the objects onthe map data which is previously stored are different from the extractedstatic objects.

Meanwhile, the image processing unit 330 may each calculate standarddeviations of a pixel value of the registered image, a pixel value ofthe obtained reference image, and pixel values of the target imagesconverted according to a homography for each of the pixels, and maydetect a dynamic static object region from the registered image based onthe calculated standard deviation. In this case, the controlling unit305 may exclude the detected dynamic object from the registered image.In addition, the controlling unit 305 may confirm position informationof the static object in the registered image from which the dynamicobject is excluded, and may update the map data based on the positioninformation of the static object.

The image processing unit 330 in FIG. 3 calculates a homography byextracting feature points between the obtained images and matching thefeature points between the respective images to calculate the matchedfeature points having common feature points. In addition, the imageprocessing unit 330 registers the images obtained by using thecalculated homography.

The image processing unit 330 according to an exemplary embodiment ofthe present invention includes all or some of a feature point extractingunit 330 a, a feature point matching unit 330 b, a homographycalculating unit 330 c, an image registration unit 330 d, a transparencyprocessing unit 330 e, and an image synthesizing unit 330 f.

Here, the image processing unit 330 may be implemented in a separatemodule using software, hardware, or a combination thereof. As anexample, according to a hardware implementation, the image processingunit 330 may be implemented using at least one of application specificintegrated circuits (ASICs), digital signal processors (DSPs), digitalsignal processing devices (DSPDs), programmable logic devices (PLDs),field programmable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, and electrical units for performingother functions.

An operation of the image processing unit 330 according to an exemplaryembodiment of the present invention will be described with reference toFIGS. 4 to 12. FIG. 4 is a diagram illustrating operations of a featurepoint extracting unit 330 a and a feature point matching unit 330 b ofan image processing unit 330 according to an exemplary embodiment of thepresent invention and FIG. 5 is a diagram illustrating a process ofmatching a reference image with a target image according to a featurepoint matching described in FIG. 4 according to an exemplary embodimentof the present invention.

FIG. 6 is a diagram illustrating a matching relationship between thereference image and the target image by a homography calculated by ahomography calculating unit 330 c of the image processing unit 330according to an exemplary embodiment of the present invention.

FIG. 7 is a diagram illustrating a distinction between a static objectand a dynamic object according to an exemplary embodiment of the presentinvention. FIGS. 8 to 11 are diagrams illustrating a transparencyprocess according to an exemplary embodiment of the present invention.

FIG. 12 is a diagram illustrating image processing results for thereference image and the target image according to an exemplaryembodiment of the present invention. The feature point extracting unit330 a extracts a plurality of feature points from each of the referenceimage and the target image determined by the controlling unit 305. Here,the plurality of feature points extracted from the reference image andthe target image, which are points at which an image brightness valuesuddenly changes in the reference image or the target image, may beedges of pixels or corner points. Specifically, an operation ofextracting the feature points by the feature point extracting unit 330 aaccording to an exemplary embodiment of the present invention will bedescribed with reference to FIG. 4.

In FIG. 4, reference numerals 405 and 410 denote images photographed atthe same place and denotes images in which the camera calibrationprocess is completed. It is assumed in FIG. 4 that the image ofreference numeral 405 and the image of reference numeral 410 areregistered, and accordingly, the image of reference numeral 410 refersto the reference image that is a reference for the image registration,and the image of reference numeral 405 refers to the target image thatis a target for the image registration.

In FIG. 4, the feature point extracting unit 330 a each extracts thefeature points from the reference image 410 and the target image 405,where reference number 415 is a diagram illustrating the feature pointsextracted from the target image 405, and reference numeral 420 is adiagram illustrating the feature points extracted from the referenceimage 410.

The feature extraction of the respective images refers to extractingcharacterized points from the respective images, and as an example,points at which the image brightness value suddenly changes, such asedges of pixels or corner points in the images may correspond to thefeature points. As algorithms for extracting the feature points of therespective images, a Harris scheme, a scale invariant feature transform(SIFT) scheme, an oriented fast and rotated brief (ORB) scheme, afeatures from accelerated segment test (FAST) scheme, and the like maybe used.

An image of reference numeral 415 illustrates feature points determinedas the feature points in the image of reference numeral 405 by applyingthe FAST scheme, and an image of reference numeral 425 illustratesfeature points determined as the feature points in the image ofreference numeral 410 by applying the FAST scheme.

Reference number 415 a illustrates one of the feature points extractedfrom reference numeral 415, and reference numeral 415 b illustrates afirst feature point group in which the plurality of feature pointsextracted from reference numeral 415 are grouped. In addition, referencenumber 420 a illustrates one of the feature points extracted fromreference numeral 420, and reference numeral 420 b illustrates a secondfeature point group in which the plurality of feature points extractedfrom reference numeral 420 are grouped.

According to an exemplary embodiment of the present invention, when thefeature point extracting unit 330 a completes the extraction of thefeature points such as reference numerals 415 and 420, the feature pointmatching unit 330 b matches the feature points of the reference imageand the target image using the first feature point group 415 b extractedfrom the reference image and the second feature point group 420 bextracted from the target image such as reference numeral 430. Thefeature point matching according to the present invention refers tomatching the feature points extracted from the respective images to eachother to find common feature points existing between the reference imageand the target image, and may use a random sample consensus (RANSAC)scheme. The RANSAC scheme is a method of selecting data in which aconsensus is maximally formed by randomly selecting sample data fromdata, and is a method capable of more accurately matching the featurepoints by acquiring the data in which error and noise are minimized inthe data.

Reference numeral 430 illustrates a matching relationship between theimage 415 and the image 420 by matching the extracted feature pointsbetween the image 415 on which the feature points extracted from thetarget image 405 are marked and the image 420 on which the featurepoints extracted from the reference image 410 are marked. Examples ofthe method of matching the feature points of the respective images mayinclude a least square method, M-estimator SAC (MSAC), maximumlikelihood SAC (MLESAC), locally optimized RANSAC (LO-RANSAC), and thelike other than the RANSAC scheme.

When the feature points of the reference image and the feature points ofthe target image are matched by the feature point matching unit 330 b,the homography calculating unit 330 c calculates a homography, which isa transformation matrix between the reference image and the targetimage, using information of matched pairs between the first featurepoint group 415 b and the second feature point group 420 b.

The homography is a transformation matrix for matching a target image ofa 2D plane to the reference image. In order to determine the homography,at least four matched pointed are required in the respective images, anda relationship matrix H between the matched points is defined as a 3×3matrix H as in the Equation 1.

$\begin{matrix}{{w\begin{bmatrix}x^{\prime} \\y^{\prime} \\1\end{bmatrix}} = {\begin{bmatrix}{h11} & {h12} & {h13} \\{h21} & {h22} & {h23} \\{h31} & {h32} & {h33}\end{bmatrix}\begin{bmatrix}x \\y \\1\end{bmatrix}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Here, x and y denote feature point coordinates of the reference image,and ‘1’ means that the reference image and the target image arehomogenous to each other. W, which is a weight, is defined as aconstant, not ‘0’. The 3×3 matrix is the transformation matrix H.

FIG. 6 is a diagram illustrating a matching relationship between thereference image and the target image according to an exemplaryembodiment of the present invention. In FIG. 6, reference numeral 600denotes a reference image, reference numeral 605 denotes a target image,and reference numeral 610 illustrates that a matching relationshipbetween a feature point matrix (x, y, 1) of the reference image and afeature point matrix (x′, y′, 1) of the target image exists using thecalculated homography matrix.

The image registration unit 330 d converts the target image 410 to beregistered to the reference image 405 using the homography calculated bythe homography calculating unit 330 c.

Specifically, in FIG. 5, reference numeral 505 denotes the referenceimage 410 in FIG. 4 and reference numeral 510 denotes an image obtainedby converting the target image of reference numeral 405 in FIG. 4 usingthe homography matrix calculated by the homography calculating unit 330c. In addition, the image registration unit 330 d registers thereference image 505 and the converted target image 510 as in referencenumeral 515 of FIG. 5. As illustrated in FIG. 5, the image registrationunit 330 d performs the registration based on the matched feature pointsof the respective images and naturally registers two different images byblending different images.

The transparency processing unit 330 e performs a transparency processfor each of the reference image and the target image which areregistered by the image registration unit 330 d.

An operation of performing the transparency process by the transparencyprocessing unit 330 e according to an exemplary embodiment of thepresent invention will be described in detail with reference to FIGS. 8to 11. The transparency process performed by the transparency processingunit 330 e according to the present invention may be performed as in amethod described in FIGS. 8 to 11. A total of ten images exist in FIG.8, a first FIG. 800 of a circular shape exists in all of the ten images,a second FIG. 810 of a triangular shape exists in five images, and athird FIG. 820 of a quadrangular shape exists in three images. In thisstate, when the respective images are converted with a predeterminedtransparency, pixel values of the first to third FIGS. 800, 810, and 820decrease as illustrated in FIG. 9, thereby causing a blurring effect.Thereafter, when the respective images which are transparency-processedare overlapped with each other, as illustrated in FIG. 10, the firstFIG. 800 has the same clear shape as before the transparency process,but the second FIG. 810 and the third FIG. 820 have lower definitionthan before the transparency process.

As described above, a method of performing a transparency process forthe target image in the transparency processing unit 330 e according toan exemplary embodiment of the present invention includes performing animage processing such that a transparency target image has a pixel valuesmaller than the pixel value of the original image by multiplying eachpixel value (0 to 255) of the transparency target image by apredetermined constant. In general, R, G, and B pixel values of eachpixel have values of 0 to 255, and when the pixel value has a smallvalue, each pixel is visually blurred, and as a result, the entire imageis also blurred.

As illustrated in FIG. 11, according to an exemplary embodiment of thepresent invention, assuming that N images are registered, when it isassumed that transparency of images for which the transparency processis not performed is ‘1’, the transparency process may be performed bymultiplying pixel values of the respective image to be transparentizedby 1/N. FIG. 9 illustrates transparency of each image of a case ofmultiplying a pixel of each image by 1/10 because the total number oftransparency target images is ten. Here, in FIG. 11, a 0-th image is thereference image, and first to N−1-th images are the target imagesconverted by the homography.

When the N images transparentized as described above are registered, thestatic objects existing in each image remain on the registered image asit is, but the dynamic objects disappear. That is, when the respectiveimages are transparentized and then registered, since the pixelcorresponding to the static object in each image is in a state in whichthe transparency multiplied by the same constant is applied, in a casein which the respective pixel values are added, the added pixel value isequal to an original pixel value or is close thereto, and as a result,the respective image also exists on the registered image as it is.

An operation of extracting the static object through the transparencyprocess according to an exemplary embodiment of the present inventionwill be again described in detail. First, a pixel value for each ofpixels of the registered image generated by registering the referenceimage and the target image converted according to the homography iscalculated. Here, the calculated pixel value for each of the pixels ofthe registered image is equal to an average value of each of pixels ofthe N registered images.

In addition, standard deviations of the calculated pixel value of theregistered image, the pixel value of the obtained reference image, andthe pixel values of the converted target images are calculated for eachof the pixels. The image processing apparatus 300 according to anexemplary embodiment of the present invention may determine a pixelhaving a value within a predetermined threshold value among thecalculated standard deviations as a pixel configuring the static object.

On the contrary, the image processing apparatus 300 according to anexemplary embodiment of the present invention may determine a pixelhaving a value exceeding a predetermined threshold value among thecalculated standard deviations as a pixel configuring the dynamicobject. With such a scheme, the image processing apparatus 300 mayclassify the static object and the dynamic object from the registeredimage by calculating the standard deviations of the pixel values of therespective image and the pixel values of the registered image.

As such, the reason that the image processing apparatus 300 may classifythe static object and the dynamic object from the average value for eachof the pixels of the registered image is because the static objectexists per each obtained image and a change of the pixel value for eachof the respective images is small, such that the standard deviation ofthe pixel values included in the static object is lower than apredetermined threshold value.

On the other hand, since the dynamic object does not exist per theobtained image but exists only in some images, a change of the pixelvalue for each of the respective images is large, and accordingly, thestandard deviation of the pixel values included in the dynamic object ishigher than the predetermined threshold value. Here, the predeterminedthreshold value may be determined by an experiment.

In addition, the background other than the roads, the traffic lights,the crosswalks, and the buildings may exist in the static objectsextracted according to an exemplary embodiment of the present invention.In this case, according to the present invention, in order to create themap, it is necessary to identify only the static objects to be reflectedto the actual road data among the extracted static objects through anadditional method, where the extracted static objects may be identifiedthrough a deep learning or a machine learning. Therefore, thecontrolling unit 305 of the image processing apparatus 300 according toan exemplary embodiment of the present invention may separate only thestatic objects such as the traffic lights, the roads, the crosswalks,and the like necessary to create or update the map data from theextracted static objects through the deep learning or the machinelearning.

In FIG. 9, in a case in which the transparency processing unit 330 emultiplies the pixel value of each image by 1/10 for the transparencyprocess of each image and registers ten images, it may be confirmed thatthe first FIG. 800 exists in the same way as before performing thetransparency process, but the second and third FIGS. 810 and 820disappear as compared to before performing the transparency process, asillustrated in FIG. 10. FIG. 10 illustrates a case in which the imageregistration for ten images is performed, but when an infinite number ofimages are overlapped, the dynamic objects having a large change of thepixel value between the images will be recognized as nonexistent.

The contents of overlapping the respective images will be describedusing the Equation 2 below.

A(t)=α(A(t−1))+(1−α)f(t), 1≤t≤N−1  [Equation 2]

Here, A(t) uses a t−1-th image and a t-th image as average imagesobtained for t time. Further, α, which is a weight, has a value of 0.5and refers to add a 50% pixel value of the t−1-th image and a 50% pixelvalue of the t-th image. Further, f(t) denotes the t-th image.

As an example, the Equation 2 will be applied as follows.

A(1)=αf(0)+(1−α)f(1),

A(2)=α(A(1))+(1−α)f(2),

A(3)=α(A(2))+(1−α)f(3)

Further, the image processing apparatus 300 may perform an imageprocessing using the images received in units of predetermined periodsuch as one day or one week, and may perform the image processing usingall of the received images or using an arbitrarily selected image. As anexample, as illustrated in FIG. 11, if there are N images from 0 to N−1obtained by photographing a specific region for one day by the first tothird vehicles 100, 110, and 120 and transmitted to the image processingapparatus 300, the image processing apparatus 300 may also extract thestatic objects by performing the transparency process and registrationfor the entirety of the images from 0 to N−1, and may also extract thestatic objects only using some of the entirety of the images.

A method of extracting, by the controlling unit 305 of the imageprocessing apparatus 300, the static objects such as the buildings, theroads, and the like from the images transparentized by the transparencyprocessing unit 330 e according to the contents described above will bedescribed.

In FIG. 7, reference numerals 740 and 750, which are images photographedat a predetermined time interval in a specific region, are imagephotographed by the cameras or smart phones mounted in the vehicles. Thephotographed images include a building 700, a road 720, and a trafficlight 730 as the static objects and include vehicles 710 as the dynamicobjects. The image processing apparatus 300 may obtain a plurality ofimages related to the specific region by classifying only the imagesrelated to the specific region using acquisition information of theimages received from the electronic apparatus 200 for a predeterminedtime (e.g., ‘one day’).

In addition, when the image processing apparatus 300 performs the imageregistration and the transparency process for the received images, thestatic objects such as the building 700, the road 720, and the trafficlight 730 exist as it is, but the dynamic objects such as the vehiclesdisappear, as illustrated by reference numeral 760. According to anexemplary embodiment of the present invention, as the number of theregistered images is increased, the static object and the dynamic objectmay be more clearly classified when the transparency process isperformed.

The image synthesizing unit 330 f performs synthesis for thetransparency-processed images and transmits the synthesized image to thecontrolling unit 305.

The storing unit 320 includes a map data storing unit 320 a and areceived image storing unit 320 b. The map data storing unit 320 a storemap data which is previously created and also store map data which isupdated by the controlling unit 305. The received image storing unit 320b stores the image received through the receiving unit 310.

The controlling unit 305 according to an exemplary embodiment of thepresent invention performs a control so that the images received throughthe receiving unit 310 are stored in the received image storing unit 320b, and updates the map data when it is necessary to update the storedmap data to store the updated map data in the map data storing unit 320a.

In addition, the controlling unit 305 extracts the static objects fromthe images transparency-processed by the image processing unit 330,compares the extracted static objects with objects on the map data whichis previously stored in the map data storing unit 320 a, and updates themap data when the objects on the map data which is previously stored andthe extracted static objects are different.

Specifically, the controlling unit 305 confirms position information ofthe extracted static objects and examines whether or not the objectsdifferent from the extracted static objects exist at positionscorresponding to the position information of the static objectsconfirmed in the map data which is previously stored. In addition, as aresult of the examination, if the different objects exist, thecontrolling unit 305 reflects the extracted static objects to the mapdata which is previously stored corresponding to the positioninformation and update the map data to store it in the map data storingunit 320 a.

The images received by the image processing apparatus 300 from thevehicles positioned on the road according to an exemplary embodiment ofthe present invention include fixed geographic features, static objectssuch as lane information marked on the road such as left turn/rightturn, and dynamic objects such as other vehicles positioned on the road,pedestrians, and the like. Therefore, it is important to excludeunnecessary dynamic objects except for the static objects from theimages received when the image processing apparatus 300 generates orupdates the map data according to an exemplary embodiment of the presentinvention.

However, since the static objects in the received images are covered bythe dynamic objects, a case in which it is difficult for the imageprocessing apparatus 300 to identify the static objects may occur.Therefore, according to an exemplary embodiment of the presentinvention, in order to solve the above-mentioned problem, the imageprocessing apparatus 300 registers the reference image and the targetimage, performs the transparency process, and then extracts the staticobjects.

FIG. 12 is a diagram illustrating an operation of extracting staticobjects using the obtained images on an actual road according to anexemplary embodiment of the present invention.

In FIG. 12, a right turn lane marking 1201 exists on a road of referencenumeral 1205 but a vehicle exists on a road of reference numeral 1210 inwhich it is difficult to extract the right turn lane marking 1201. Inthis case, when the image processing is performed according to anexemplary embodiment of the present invention, the vehicle, which is thedynamic object, appears to be dim, but the extraction of the right turnlane marking 1201, which is the static object, may be easily identified,as in reference numeral 1215. Therefore, according to an exemplaryembodiment of the present invention, since the image processingapparatus 300 may confirm the right turn marking 1201 existing on theroad, it may update the map data so as to reflect the correspondingcontents when the right turn marking 1201 does not exist in the map datawhich is previously created.

The transmitting unit 315 transmits the map data stored in the storingunit 320 or the map data updated by the controlling unit 305 to theelectronic apparatus 200 of the vehicle or the smartphone of the user.Further, the transmitting unit 315 may also transmit a messagerequesting a transmission of images obtained at a current position tothe electronic apparatuses 200 of the vehicles.

Further, the exemplary embodiment of the present invention describesthat the transparency processing unit 320 e performs the transparencyprocess after the image registration is performed by the imageregistration unit 320 d, but is not limited thereto.

According to another exemplary embodiment of the present invention, thetransparency processing operation may also be performed afterdetermining the images to be registered by the controlling unit 305, mayalso be performed after the feature points are extracted, and may alsobe performed after the homography is calculated.

FIG. 13 is an operation flowchart of an image processing apparatusaccording to an exemplary embodiment of the present invention.

The image processing apparatus 300 receives images in an operation(S1305) and stores the received images in the storing unit in anoperation (S1310). In addition, the image processing apparatus 300determines the reference image and the target image in an operation(S1315) and performs an image pre-processing in an operation (S1320).The image pre-processing operation includes an image processing to unifythe color, brightness, and definition of the image so as not to beinfluenced by recognition of the image and a synthesizing process withother images, and mainly uses functions such as Curves, Levels,Brightness/Contrast, Color Balance, Shadows/Highlight, and the like.

If the image pre-processing operation is completed, the image processingapparatus 300 performs an image processing operation according to anexemplary embodiment of the present invention in an operation (S1325).The image processing operation performed in the operation (S1325) willbe descried with reference to FIG. 14.

If the image processing in the operation (S1325) is completed, the imageprocessing apparatus 300 extracts static objects from the registeredimage in an operation (S1330) and compares the extracted static objectswith the map data which is previously store in an operation (S1335).Specifically, the image processing apparatus 300 confirms whether or notobjects different from the extracted static objects exist at a positioncorresponding to position information of the static object confirmed inthe map data which is previously stored.

As a result of the confirmation of the operation (S1335), if the objectsdifferent from the static objects extracted from the map data which ispreviously stored exist in an operation (S1340), the image processingapparatus 300 update the map data in an operation (S1345) and transmitsthe updated map data to the electronic apparatus or the smartphone ofthe user in an operation (S1350).

FIG. 14 is a flowchart illustrating the operation (S1325) of FIG. 13 indetail.

The image processing apparatus 300 extracts first feature points fromthe reference image and extracts second feature points from the targetimage in an operation (S1405). In addition, the image processingapparatus 300 matches a first feature point group in which the firstextracted feature points are grouped and a second feature point group inwhich the second extracted feature points are grouped in an operation(S1410), and calculates a homography using information of matched pairsbetween the first feature point group and the second feature point groupin an operation (S1415).

The image processing apparatus 300 converts the target image using thehomography calculated in the operation (S1415) in an operation (1420),and registers the reference image and the converted target image in anoperation (S1425).

In addition, the image processing apparatus 300 performs thetransparency process for the registered images in an operation (S1430)and then synthesizes the transparency-processed images in an operation(S1435). The operation (S1435) is an optional operation and may not alsobe performed as needed. That is, when the image processing apparatus 300extracts the static objects from the registered images according to anexemplary embodiment of the present invention, the image processingapparatus 300 may also extract the static objects in a state in whichthe transparency process is completed and may also extract the staticobjects after the transparency process is performed and the imagesynthesis is also completed.

FIG. 15 is a block configuration diagram of an image processingapparatus for creating a map according to another exemplary embodimentof the present invention. An input unit 1550 may receive a command forperforming an operation of an image processing apparatus 1500 from theuser, and may include a keypad, a touch pad (resistive/capacitive), andthe like.

An image processing unit 1510 processes images received from thevehicles positioned on the road and extracts a variety of informationsuch as fixed geographic features, extending/closing of the road, orlane information such as a left turn/right turn from the receivedimages.

A communicating unit 1520 performs communication with electronicapparatuses or mobile terminals such as smartphones included in thevehicles to receive the photographed images or transmit created map datato the electronic apparatuses or the mobile terminals. The communicatingunit 1520 may perform communication in a wired communication scheme aswell as various wireless communication schemes such as Bluetooth, Wi-Fi,wireless broadband, 3rd generation (3G), WCDMA scheme, long termevolution (LTE), and a 4th generation (4G) communication schemes.

A storing unit 1530 stores the images received by the electronicapparatuses or the mobile terminals such as the smartphones included inthe vehicles, map data generated by a map creating server (not shown) orthe image processing apparatus 1500, and instructions which areexecutable by the controlling unit 340 of the image processing apparatus1500. As the storing unit 1530, various storing mediums such as a randomaccess memory (RAM), a static random access memory (SRAM), a read onlymemory (ROM), an electrically erasable programmable read-only memory(EEPROM), a programmable read-only memory (PROM), a magnetic memory, amagnetic disk, an optical disc, and the like may be used.

A controlling unit 1540 performs a control to process the received imageto generate the map data, or generate updated data for updating the mapdata, and transmit the generated map data to the electronic apparatusesor the mobile terminals such as the smartphones included in the vehiclesthrough the communicating unit 1520.

FIG. 16 is a method flowchart illustrating a method for creating a mapof the image processing apparatus of FIG. 15.

FIG. 16 relates to a case of updating map data which is previouslystored when driving paths of the vehicles driving on the road aredifferent from the map data which is previously stored. As an example, acase in which the road is a left turn no region on the map data which ispreviously stored, but the vehicle turns left, a case in which thevehicle is being driven even though there is no road in the map datawhich is previously stored, or the like may correspond to such asituation. Here, since the road or the lane information marked on theroad corresponds to the static object, the map which is previouslystored may be modified by the image processing method as describedabove.

Specifically, the image processing apparatus 1500 receives and stores avariety of data such as the images received from the electronicapparatuses of the vehicles driving on the road, user driving logs, andthe like in an operation (S1600). Next, in an operation (S1610), thecontrolling unit 1540 of the image processing apparatus 1500 confirmswhether or not there are different points between the map data such asthe road, the lane information, and the like stored in the map datawhich is previously stored and the user driving contents by analyzingthe user driving logs.

If there are the different points in an operation (S1620) (Yes inS1620), the controlling unit 1540 controls the image processingapparatus 1510 so as to perform the image processing for thecorresponding point and extracts static objects such as the road and thelane information for the corresponding point in an operation (S1630).Here, it is preferred that the image processing apparatus 1500 performsthe image processing operation when the images are obtained from theelectronic apparatuses of the plurality of vehicles which are driven atthe corresponding point.

Next, the controlling unit 1540 compares the extracted static objectswith the map data to confirm whether to necessary to modify data for thestatic objects included in the map data in an operation (S1640), andgenerates update information for modifying the map data when it isnecessary to modify the data for the static objects in an operation(S1650). The controlling unit 1540 modifies the map data using theupdate information generated in the operation (S1650) in an operation(S1660) and transmits the update information generated in the operation(S1660) to the electronic apparatuses of the vehicles which are drivenon the road. Here, the controlling unit 1540 may transmit the map datato the electronic apparatus capable of using the map data such as thesmartphone or the like through the communicating unit 1520 so that theuser may conveniently use the map data.

FIG. 17 is a block configuration diagram of an electronic apparatusaccording to still another exemplary embodiment of the presentinvention.

An input unit 1715 transmits a variety of control commands input fromthe user to a controlling unit 1705, and may be a touch screen panel, akeyboard, or a mouse. A communicating unit 1720 performs communicationwith the image processing apparatus according to an exemplary embodimentof the present invention or other electronic apparatuses. An output unit1725, which is a configuration for providing information to the user insound or visual way, may be a display or a speaker. A camera unit 1730photographs subjects by a control of the controlling unit 1705.

The image processing unit 1710 may perform image processing such asfeature point extraction, transparency process, or image registrationfor the images photographed by the camera unit 1730 as described in thepresent specification to allow the controlling unit 1705 to extract thestatic objects.

A storing unit 1735 stores instructions and map data which may beperformed by the controlling unit 1705 so that the electronic apparatus1700 may be operated.

In a case in which the vehicle in which the electronic apparatus 1700according to another exemplary embodiment of the present invention ismounted drives on the road, the controlling unit 1705 performs a controlso that the images are obtained through the camera unit 1730 and loadsthe map data stored in the storing unit 1735. In addition, thecontrolling unit 1705 performs a path guidance through the output unit1725 using the loaded map data and a position of the vehicle which iscurrently measured, and extracts the static objects by performing theimage processing for the images obtained by the camera unit 1730 by theimage processing unit 1710.

In addition, the controlling unit 1705 compares the extracted staticobjects with the map data, examines whether or not the extracted staticobjects exist in the map data used for the path guidance, and request anupdate of the map data stored in the storing unit 1735 to the imageprocessing apparatus through the communicating unit 1720 when theextracted static objects do not exist in the map data.

If the updated map data is received through the communicating unit 1720,the controlling unit 1735 updates the map data which is previouslystored in the storing unit 1735.

FIG. 18 is a flowchart of an operation of the electronic apparatus 1700of FIG. 17.

If the vehicle drives on the road (S1805), the electronic apparatus 1700installed in the vehicle obtains images in an operation (S1810) andloads map data in an operation (S1815). In addition, the electronicapparatus 1700 performs a path guidance using the loaded map data and aposition of the vehicle which is currently measured in an operation(S1820), and extracts static objects by performing an image processingfor the obtained images in an operation (S1825).

The electronic apparatus 1700 compares the extracted static objects withthe map data in an operation (S1830) and examines whether or not theextracted static objects exist in the map data used for the pathguidance in an operation (S1835). As a result of the examination of theoperation (S1835), if the static object different from the extractedstatic objects exist in the map data, the electronic apparatus 1700requests an update of the map data which is previously stored to theimage processing apparatus in an operation (S1840). If the updated mapdata is received in an operation (S1845), the electronic apparatus 1700updates the map data which is previously stored in an operation (S1850).

FIGS. 19 and 20 are diagrams illustrating a transparency process forfive images according to an exemplary embodiment of the presentinvention.

FIG. 19 illustrates a total of five images (reference numerals 1905,1910, 1915, 1920, and 1925) obtained at the same point according to anexemplary embodiment of the present invention and reference numerals2005, 2010, 2015, 2020, and 2025 of FIG. 20 illustrate results obtainedby performing a transparency process for each of the images of referencenumerals 1905, 1910, 1915, 1920, and 1925 of FIG. 19. Since the totalnumber of images to be targeted to the transparency process is 5 (N=5)in FIGS. 19 and 20, a constant multiplied with a pixel value of each ofthe images of reference numerals 1905, 1910, 1915, 1920, and 1925 ofFIG. 19 to perform the transparency process according to an exemplaryembodiment of the present invention is ⅕. That is, the respective imagesof FIG. 20 are the transparency-processed images obtained by multiplyingthe respective pixel values of the respective images of FIG. 19 by ⅕.

In addition, FIG. 21 is a diagram illustrating an image registration ofthe transparency-processed images of FIG. 20 and FIG. 22 is a diagramillustrating a result of the image registration of FIG. 21, where it maybe seen that the vehicles, which are the dynamic objects, disappear,while roads, lane marking lines, crosswalks, traffic lights, and thelike, which are static objects, exist as it is. According to the presentinvention, the static objects may be extracted from the images in thismethod and used to update or create the map data.

Meanwhile, the above-mentioned example illustrates that the transparencyprocessing unit performs the transparency process by multiplying therespective pixel values (0 to 255) of the transparency target images bya predetermined constant and performing the image processing so that therespective pixel values (0 to 255) of the transparency target imageshave a pixel value smaller than the original pixel values of the images,but is not limited thereto. According to another implementation of thepresent invention, the transparency processing unit may adjusttransparency of the corresponding image by adjusting an ALPHA (A) valuecorresponding to a transparency level in an RGBA value of thetransparency target image. Here, the Alpha (A) value may be defined inadvance and may also be updated periodically.

Further, the above-mentioned example illustrates that the static objectsand the dynamic objects are extracted from the registered image bycalculating the standard deviations of the pixel value of the registeredimage, the pixel value of the reference image, and the pixel value ofthe target image for each of the pixels, determining the pixels in whichthe calculated standard deviation is the predetermined value or less asthe pixels for the static objects, and determining the pixels in whichthe calculated standard deviation exceeds the predetermined value as thepixels for the dynamic objects, but is not limited thereto. According toanother implementation of the present invention, the static objects andthe dynamic objects may also be classified by comparing an ALPHA (A)value in an RGBA value of the registered image with a predeterminedtransparency value.

According to the various exemplary embodiments of the present inventiondescribed above, it is possible to extract static objects positioned onthe road through the images obtained by the cameras installed in thevehicles positioned on the road and to accurately and quickly create themap using the extracted static object.

Further, according to the various exemplary embodiments of the presentinvention described above, it is possible to remotely update the mapdata in real time by receiving the images obtained through the camerasof the vehicles positioned on the road in real time, unlike an existingmap creating system through a survey.

Further, according to the various exemplary embodiments of the presentinvention described above, it is possible to provide the map service towhich the newest road environment is applied to the users bytransmitting the map data which is updated in real time to the vehiclespositioned on the road.

FIG. 23 shows a simplified block diagram of an electronic deviceaccording to various embodiments.

Referring now to FIG. 23, an electronic device 2300 may be associatedwith the electronic device 200 shown in FIG. 2. The electronic device2300 may include at least one of the elements included in the electronicdevice 200, such as e.g., the input unit 210, the communicating unit230, the camera unit 240, the display unit 220, and the display unit220. The electronic device 2300 may be incorporated into a vehicle. Forexample, the vehicle may be stationary or in motion. In the followingdescription, for convenience of description, the electronic device 2300incorporated into the vehicle may be described in the form of a vehicle.

The electronic device 2300 may include a processor 2310, a communicationcircuitry 2320, a camera 2330, a sensing circuit 2340, a memory 2350,and/or a display 2360.

The processor 2310 may be associated with the control unit 250 of FIG.2. The processor 2310 may operate to control the overall operation ofthe electronic device 2210. The processor 2310 may execute applicationsthat provide an advertisement service, an Internet service, a gameservice, a multimedia service, and/or a navigation (or map) service. Invarious embodiments, the processor 2310 may include a single processorcore or may include multiple processor cores. For example, the processor2310 may include a multi-core such as e.g., a dual-core, a quad-core, ahexa-core or the like. According to an embodiment, the processor 2310may further include a cache memory located inside or outside of it.

The processor 2310 may receive instructions from other elements of theelectronic device 2300, analyze the received instructions, and performcalculations or process data according to the analyzed instructions.

The processor 2310 may process data or signals generated or providedfrom an application. For example, the processor 2310 may requestinstructions, data, or signals from the memory 2350 to execute orcontrol the application. The processor 2310 may operate to write (orstore) or update the instructions, data, or signals into the memory 2350to execute or control the application.

The processor 2310 may analyze and process messages, data, instructions,or signals received from the communication circuitry 2320, the camera2330, the sensing circuit 2340, the memory 2350, or the display 2360.Further, the processor 2310 may generate new messages, data,instructions, or signals based on the received messages, data,instructions, or signals. The processor 2310 may provide the processedor generated messages, data, instructions, or signals to thecommunication circuitry 2320, the camera 2330, the sensing circuit 2340,the memory 2350, the display 2360, or the like.

An entirety or a portion of the processor 2310 may be electrically oroperably coupled with or connected to any other elements in theelectronic device 2300 such as e.g., the communication circuitry 2320,the camera 2330, the sensing circuit 2340, the memory 2350, or thedisplay 2360.

According to an embodiment, the processor 2310 may include one or moreprocessors. For example, the processor 2310 may include an applicationprocessor (AP) to control an upper layer of program such as e.g., anapplication program, a graphics processing unit (GPU) for configuring ascreen displayed on the display 2360 and controlling the screen, animage signal processor to control the camera 2330, a sensor hub tocontrol the sensing circuit 2340, a communication processor (CP) tocontrol the communication circuitry 2320, or the like.

The communication circuitry 2320 may be related to the receiving unit310 or the transmitting unit 320 shown in FIG. 2. The communicationcircuitry 2320 may be used to establish a communication path between theelectronic device 2300 and another electronic device (e.g., a server, anexternal electronic device, or any device embedded in a vehicle). Thecommunication circuitry 2320 may support a predetermined protocolcapable of connecting to the other electronic device in either wired orwireless connection. For example, the communication circuitry 2320 mayinclude a module (or circuit) for at least one of Bluetoothcommunication technique, BLE (Bluetooth low energy) communicationtechnique, Wi-Fi (Wireless-Fidelity) communication technique, cellular(or mobile) communication technique, or a wired communication technique.As another example, the communication circuitry 2320 may include anHDMI, a USB interface, an SD card interface, or an audio interface,being operable in association with a connection terminal such as, forexample, a high definition multimedia interface (HDMI) connector, auniversal serial bus (USB) connector, an SD card connector, an audioconnector (e.g., a headphone connector) or the like.

The communication circuitry 2320 may include a communication circuitryfor global positioning system (GPS) (or GNSS). The communication module210 may transmit/receive GPS signals. Depending on regions or bandwidthsin use, the GPS may include at least one of GLONASS (global navigationsatellite system), Beidou Navigation Satellite System (hereinafter,referred to as “Beidou”), QZSS (quasi-zenith satellite system), IRNSS(Indian regional satellite system) or Galileo (the European globalsatellite-based navigation system).

The communication circuitry 2320 may provide the processor 2310 withinformation or data received from the other electronic device throughthe communication path. The communication circuitry 2320 may transmitinformation or data provided from the processor 2310 to the otherelectronic device through the communication path.

The camera 2330 may capture a still image or a moving image. In variousembodiments, the camera 2330 may include at least one of one or morelenses (e.g., a lens assembly), an image sensor, a flash, an imagestabilizer, a buffer memory or the like. For example, the one or morelenses may collect light emitted from a subject for an image to becaptured.

According to an embodiment, the camera 2330 may include a plurality ofcameras. For example, the camera 2330 may include a first camera and asecond camera. The first camera and the second camera may be configuredat different positions of the electronic device 2300, and may be used tocapture images of external objects in different directions.

According to an embodiment, the camera 2330 may include a plurality oflens assemblies. For example, the plurality of lens assemblies may havethe same lens properties (e.g., angle of view, focal length, auto-focus,f number, or optical zoom). For example, at least one of the pluralityof lens assemblies may have the lens properties that are different fromthose of at least one of the plurality of lens assemblies. For example,at least one of the plurality of lens assemblies may be configured for awide-angle lens, and at least another of the plurality of lensassemblies may be configured for a telephoto lens.

In various embodiments, the flash may emit a light source that is usedto enhance light given off from a subject. For example, the flash mayinclude one or more light emitting diodes (e.g., a red-green-blue (RGB)LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenonlamp.

In various embodiments, the image sensor may convert light transmittedfrom a subject through one or more lenses into an electrical signal toobtain an image corresponding to the subject (e.g., an image related tothe vehicle equipped with the electronic device 2300). In an embodiment,the image sensor may include one image sensor selected from imagesensors having different properties (such as e.g., an RGB sensor, a BW(black and white) sensor, an IR sensor or a UV sensor), a plurality ofimage sensors having the same properties, or a plurality of imagesensors having different properties. Each image sensor included in theimage sensors may be implemented with, for example, a charged coupleddevice (CCD) sensor or a complementary metal oxide semiconductor (CMOS)sensor.

In various embodiments, the image stabilizer may, in response to amovement of the camera 2330 or the electronic device 2300, may move oradjust the one or more lenses or the image sensor in a certain direction(e.g., adjusting read-out timing, etc.) to at least partiallycompensates for any negative effects (such as e.g., shaking image) thatmight be caused by a movement of the captured image. According to anembodiment, the image stabilizer may be implemented with an opticalimage stabilizer, and a gyro sensor (e.g., the sensing circuit 2340) oran acceleration sensor (e.g., the sensing circuit 2340) disposed eitherinside or outside the electronic device 2300 or the camera 2330 may beused to detect the movement.

In various embodiments, the buffer memory may at least temporarily storeat least part of the image captured through the image sensor for a nextimage processing operation. For example, when a delay in imageacquisition according to a shutter or a high-speed acquisition of aplurality of images is performed, the obtained original image (e.g.,high-resolution image) may be stored in the buffer memory, and a copyimage (e.g., low-resolution image) may be previewed via the display2360. When a predetermined condition is satisfied after such a preview(e.g., by a user input or a system command), at least part of theoriginal image stored in the buffer memory may be obtained and processedby the image signal processor. In an embodiment, the buffer memory maybe configured with at least a part of the memory 2350 or a separatememory operated independently of the memory 2350.

The memory 2350 may store instructions, control instruction codes,control data or user data, for controlling the electronic device 2300.For example, the memory 2350 may include at least one application, anoperating system (OS), a middleware, and/or a device driver.

The memory 2350 may include one or more of a volatile memory, anon-volatile memory and so on. The volatile memory may include a dynamicrandom access memory (DRAM), a static RAM (SRAM), a synchronous DRAM(SDRAM), a phase-change RAM (PRAM), a magnetic RAM (MRAM), a resistiveRAM (RRAM), and a ferroelectric RAM (FeRAM). The non-volatile memory mayinclude a read-only memory (ROM), a programmable ROM (PROM), anelectrically programmable ROM (EPROM), an electrically erasableprogrammable ROM (EEPROM), a flash memory, and the like.

The memory 2350 may include non-volatile media such as e.g., a hard diskdrive (HDD), a solid-state disk (SSD, solid state disk), an embeddedmulti-media card (eMMC), a universal flash storage (UFS).

The sensing circuit 2340 may generate an electrical signal or a datavalue corresponding to an internal operation status (e.g., power ortemperature) within the electronic device 2300 or an externalenvironmental state outside the electronic device 2300. For example, thesensing circuit 2340 may include a radar sensor, a lidar sensor, agesture sensor, a gyro sensor, a barometric pressure sensor, a magneticsensor, an acceleration sensor, a speed sensor (or speedometer), a gripsensor, and a proximity sensor, a color sensor, an infrared (IR) sensor,a biometric sensor, a temperature sensor, a humidity sensor, anilluminance sensor, or the like.

In various embodiments, the sensing circuit 2340 may include atransmitter configured to emit a signal (or pulse) and a receiverconfigured to receive a reflected signal for the signal.

For example, the sensing circuit 2340 may emit a signal (e.g., light)under the control of the processor 2310 and receive its reflectedsignal. The sensing circuit 2340 may analyze the time up until thereflected signal is received, a phase shift of the reflected signal, apulse power of the reflected signal and/or a pulse width of thereflected signal, and the like, so as to identify the externalenvironment surrounding the electronic device 2300. The sensing circuit2340 may transmit information on the identified environment to theprocessor 2310.

For example, the sensing circuit 2340 may measure the time from when thesignal is emitted from the transmitter until the signal is reflected andthen received by the receiver so as to obtain distance information fromthe electronic device 2300 to the object. For example, the sensingcircuit 2340 may include a radar sensor and/or a lidar sensor.

The display 2360 may output contents, data, or signals. In variousembodiments, the display 2360 may display an image signal processed bythe processor 2310. For example, the display 2360 may display a capturedor still image. As another example, the display 2360 may display amotion picture or a camera preview image. As still another example, thedisplay 2360 may display a graphic user interface (GUI) so that the usercan interact with the electronic device 2300.

The display 2360 may be configured with a liquid crystal display (LCD)or an organic light emitting diode (OLED).

According to an embodiment, the display 2360 may be coupled with asensor capable of receiving a touch input or the like to be configuredwith an integrated type of touch screen.

In various embodiments, at least one of the communication circuitry2320, the camera 2330, or the sensing circuit 2340 may be disposedoutside the electronic device 2300.

FIG. 24 shows a simplified block diagram of a server according tovarious embodiments.

Referring then to FIG. 24, the server 2400 may include a processor 2410,a communication circuitry 2420 and/or a memory 2430. In variousembodiments, the server 2400 may operate to perform some or all of thefunctions of the electronic device 2300 of FIG. 23.

The server 2400 may be associated with the image processing apparatus300 of FIG. 3. The processor 2410 may be associated with the controlunit 305 and/or the image processing unit 330 of FIG. 3. Thecommunication circuitry 2420 may be associated with the receiving unit310 and/or the transmitting unit 315 of FIG. 3. The memory 2430 may berelated to the storage unit 320 of FIG. 3.

According to an embodiment, the server 2400 may obtain an image from theelectronic device 2300 and analyze the image. The server 2400 mayidentify information on a fixed external object located out of theelectronic device 2300, based on the image obtained from the electronicdevice 2300, and identify information on a geographic location of thefixed external object.

In various embodiments, the server 2400 may receive information on anexternal object from the electronic device 2300 of FIG. 23 and addcontent corresponding to the external object to an electronic map. Theserver 2400 may transmit data about the electronic map with the contentbeing added thereto, to the electronic device 2300.

Various embodiments described below may be performed through theelectronic device 2300 illustrated in FIG. 23 and the server 2400illustrated in FIG. 24.

FIGS. 25 and 26 each show a diagram for explaining an embodiment foridentifying a geographic location of a fixed external object accordingto various embodiments of the present disclosure.

Referring first to FIG. 25, the electronic device 2300 may beincorporated into a moving vehicle. The electronic device 2300 in thevehicle may move in a first direction 2510. For example, the electronicdevice 2300 in the vehicle may move from a first location to a secondlocation, wherein the first location may be of a position at a firsttiming point of the vehicle moving in the first direction 2510, and thesecond location may be of a position at a second timing point of thevehicle moving in the first direction 2510. The first timing may beearlier than the second timing on the basis of time.

The electronic device 2300 may capture a first image corresponding tothe exterior of the vehicle through the camera 2330 at the firstlocation. The electronic device 2300 may identify a first visual objectcorresponding to a fixed external object of a plurality of visualobjects included in the first image.

For example, the first image may include a plurality of visual objects.The electronic device 2300 may obtain the first image including theplurality of visual objects corresponding to a plurality of externalobjects outside the vehicle through the camera 2330. The plurality ofexternal objects may be fixed or movable. For example, the plurality ofvisual objects respectively corresponding to the plurality of externalobjects may include the aforementioned stationary object or dynamicobject.

According to an embodiment, the electronic device 2300 may identifywidths of the plurality of visual objects. The electronic device 2300may identify a plurality of external objects corresponding to each ofthe plurality of visual objects, based on the widths of the plurality ofvisual objects.

For example, the electronic device 2300 may store information on anaverage width of roadside trees in the memory 2350. The electronicdevice 2300 may obtain a visual object corresponding to an externalobject. The electronic device 2300 may identify the width of theexternal object, based on the width of the visual object. The electronicdevice 2300 may identify that the external object is a roadside tree bycomparing the width of the external object with the average width of theroadside trees. For example, the electronic device 2300 may receiveinformation on the average width of the roadside trees from an externalserver (e.g., the server 2400) and store the information on the averagewidth of the roadside trees into the memory 2350. As another example,the electronic device 2300 may perform a learning for the width of theroadside tree. The electronic device 2300 may obtain information on theaverage width of those roadside trees, based on the learning.

According to an embodiment, the electronic device 2300 may identify thewidths of the plurality of visual objects as well as various informationof the plurality of visual objects (e.g., color, shape, type, and/orsize, etc.). The electronic device 2300 may identify a plurality ofexternal objects respectively corresponding to the plurality of visualobjects, based on various information of the plurality of visualobjects. For example, the electronic device 2300 may identify the shapeof one of the plurality of visual objects. The electronic device 2300may identify an external object corresponding to the one visual objectas a traffic light, based on the shape of the one visual object being ofthe traffic light shape.

Meanwhile, while the electronic device 2300 is located in the firstlocation, the electronic device 2300 may identify a first virtual planespaced apart from a center point of the lens of the camera 2330 by afocal length of the lens in a first direction 2510. For example, thefirst virtual plane may be formed perpendicular to the first direction2510.

The electronic device 2300 may identify, at the first location, a firstintersection point between a first straight line extending from thecenter point of the lens towards a fixed external object and the firstvirtual plane.

After moving from the first location in the first direction 2510, theelectronic device 2300 may obtain a second image corresponding to theexterior of the vehicle at a second location, through the camera 2330.The electronic device 2300 may identify a second visual objectcorresponding to the fixed external object of a plurality of visualobjects included in the second image. For example, the electronic device2300 may identify the fixed external object based on the first image andthe second image, being obtained at different positions.

While the electronic device 2300 is located in the second location, theelectronic device 2300 may identify a second virtual plane spaced apartfrom the center point of the lens of the camera 2330 by the focal lengthof the lens, in the first direction 2510.

The electronic device 2300 may identify, at the second location, asecond intersection point between a second straight line extending fromthe center point of the lens towards the fixed external object and thesecond virtual plane.

The electronic device 2300 may determine a geographic location of thefixed external object, based on the first intersection point and thesecond intersection point. The geographic location of the fixed externalobject may imply an absolute location. For example, the geographiclocation of a fixed external object may be represented in latitude andlongitude.

According to an embodiment, the electronic device 2300 may identify adistance between the first location and the second location in order todetermine the geographic location of the fixed external object.

For example, the electronic device 2300 may obtain a speed of theelectronic device 2300 between the first location and the secondlocation. The electronic device 2300 may identify the time taken to movefrom the first location to the second location. The electronic device2300 may identify a distance between the first location and the secondlocation, based on the speed of the electronic device 2300 between thefirst location and the second location and the time taken to move fromthe first location to the second location.

As another example, the electronic device 2300 may identify the firstlocation of the electronic device 2300 based on the GPS communicationcircuitry. In other words, the electronic device 2300 may identify thefirst location of the electronic device 2300 as an absolute location,based on the GPS communication circuitry. That is, the electronic device2300 may identify the latitude and longitude of the first location,based on the GPS communication circuitry.

The electronic device 2300 may identify the second location of theelectronic device 2300 based on the GPS communication circuitry. Inother words, the electronic device 2300 may identify the second locationof the electronic device 2300 as an absolute location, based on the GPScommunication circuitry. That is, the electronic device 2300 mayidentify the latitude and longitude of the second location, based on theGPS communication circuitry. The electronic device 2300 may identify adistance between the first location and the second location, based onthe identified first location and second location.

In the meantime, the electronic device 2300 may determine the geographiclocation of the fixed external object, based on the first intersectionpoint, the second intersection point, and the distance between the firstlocation and the second location. In other words, the electronic device2300 may determine a relative location of the fixed external object onthe basis of the second location, based on the first intersection pointand the second intersection point. The electronic device 2300 may obtainthe geographic location of the second location based on the GPScommunication circuitry, and on the basis of the obtained geographiclocation of the second location and the second location, may determinethe geographic location of the fixed external object, based on therelative location of the fixed external object.

The electronic device 2300 may transmit information on the geographiclocation of the fixed external object to the server 2400, in order toadd a content corresponding to the fixed external object to theelectronic map. In other words, the information on the geographiclocation of the fixed external object may be transmitted to the server2400, in order to add the content corresponding to the fixed externalobject to the electronic map.

Hereinafter, a more detailed technical feature of determining ageographic location of a fixed external object will be described withreference to FIG. 26.

Referring to FIGS. 25 and 26, the electronic device 2300 may determine ageographic location 2630 of a fixed external object, based on a firstintersection point 2610 and a second intersection point 2620. A processfor determining the geographic location 2630 of the fixed externalobject will be described in further detail, hereinafter.

While the electronic device 2300 is located in the first location, theelectronic device 2300 may identify a first virtual plane 2611 spacedapart from a center point 2601 of the lens of the camera 2330 by a focallength 2612 of the lens in a first direction 2510.

In the first location, the electronic device 2300 may identify a firstintersection point 2610 between a first straight line extending from thecenter point 2601 of the lens towards the fixed external object and thefirst virtual plane 2611. The electronic device 2300 may identify athird intersection point 2613 between a third straight line extending inthe first direction 2510 from the center point 2601 of the lens and thefirst virtual plane 2611. The electronic device 2300 may identify adistance 2614 between the first intersection point 2610 and the thirdintersection point 2613.

While the electronic device 2300 is located in the second location, theelectronic device 2300 may identify a second virtual plane 2621 spacedapart from the center point 2602 of the lens of the camera 2330 by thefocal length 2622 of the lens in the first direction 2510.

In the second location, the electronic device 2300 may identify a secondintersection point 2620 between a second straight line extending fromthe center point 2602 of the lens towards the fixed external object andthe second virtual plane 2621. The electronic device 2300 may identify afourth intersection point 2623 between a third straight line extendingin the first direction 2510 from the center point 2602 of the lens andthe second virtual plane 2621. The electronic device 2300 may identify adistance 2624 between the second intersection point 2620 and the fourthintersection point 2623.

The electronic device 2300 may identify a distance 2640 between thecenter point 2601 of the lens and the center point 2602 of the lens. Thedistance 2640 may correspond to a distance that the electronic device2300 moved from the first location to the second location. For example,the distance 2640 may be identified based on the GPS communicationcircuitry of the electronic device 2300. As another example, thedistance 2640 may be identified based on a speed between the firstlocation and the second location and the time taken to move from thefirst location to the second location.

The electronic device 2300 may determine the relative location of thefixed external object on the basis of the second location, based on thedistance 2614, the distance 2624, and the distance 2640.

On the basis of the second location, a distance 2650 spaced apart in thefirst direction 2510 of the fixed external object can be set as shown inthe following Equation 3.

$\begin{matrix}{\mspace{79mu}{{Z = \frac{B\text{?}}{x_{l} - x_{r}}}{\text{?}\text{indicates text missing or illegible when filed}}}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

Referring to Equation 3, ‘Z’ is a distance 2650 by which thegeographical location 2630 of the fixed external object is spaced apartin the first direction 2510 on the basis of the second location. ‘B’ isthe distance 2640, ‘x_(r)’ is the distance 2614, and ‘x_(l)’ is thedistance 2624.

On the basis of the second location, a distance 2660 spaced apart in asecond direction 2520 perpendicular to the first direction 2510 of thefixed external object can be set as in the Equation 4 below.

$\begin{matrix}{\mspace{79mu}{{X = \frac{B\text{?}}{\text{?}}}{\text{?}\text{indicates text missing or illegible when filed}}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Referring to the Equation 4, ‘X’ is a distance 2660 by which thegeographic location 2630 of the external object is spaced apart in thesecond direction 2520 perpendicular to the first direction 2510, on thebasis of the second location. ‘B’ is the distance 2640, ‘x_(r)’ is thedistance 2614, and ‘x_(l)’ is the distance 2624. F is the focal length(2612, 2622) of the lens of camera 2330.

Meanwhile, the electronic device 2300 may identify the geographiclocation of the second location through the GPS communication circuitry.For example, the electronic device 2300 may identify the latitude andlongitude of the second location.

The electronic device 2300 may identify the geographic location of thefixed external object, based on the geographic location of the secondlocation and the relative location of the fixed external object on thebasis of the second location.

According to an embodiment, at least part of the above-describedoperation of the electronic device 2300 may be performed in the server2400.

For example, the electronic device 2300 may transmit the first imageobtained at the first location and the second image obtained at thesecond location, to the server. The server 2400 may receive the firstimage and the second image from the electronic device 2300. The server2400 may identify a first visual object corresponding to a fixedexternal object of a plurality of visual objects. For example, theserver 2400 may respectively classify the plurality of visual objectsinto one of a stationary object and a dynamic object. The server 2400may identify the first visual object corresponding to the fixed externalobject among at least one visual object classified as a stationaryobject of the plurality of visual objects.

In the meantime, the server 2400 may identify at least a portion of datarelated to a shape of the fixed external object, data related to a typeof the fixed external object, and data related to a function of thefixed external object, based on a machine learning model.

For example, the server 2400 may receive, from the electronic device2300, additional information (e.g., information on the geographiclocation of the second location) for obtaining information on thegeographic location of the fixed external object through the first imageand the second image. The server 2400 may obtain the information on thegeographic location of the fixed external object based on the additionalinformation. For example, the server 2400 may obtain the information onthe geographic location of the fixed external object, by performing thesame or similar operation as the above-described operation of theelectronic device 2300.

FIGS. 27 and 28 each show a diagram for explaining an embodiment foridentifying a geographic location of a fixed external object accordingto various embodiments of the present disclosure.

Referring now to FIG. 27, the electronic device 2300 in the vehicle maymove on in the first direction 2730. The electronic device 2300 mayinclude at least two cameras. For example, the electronic device 2300may include a first camera 2710 and a second camera 2720, wherein thesecond camera 2720 may be spaced apart from the first camera 2710 by afirst distance “d”.

The electronic device 2300 may obtain an image corresponding to theexterior of the vehicle through at least one of the first camera 2710and the second camera 2720. For example, the electronic device 2300 maycapture an image corresponding to the exterior of the vehicle througheither one of the first camera 2710 and the second camera 2720. Asanother example, the electronic device 2300 may obtain an imagecorresponding to the exterior of the vehicle, by synthesizing the firstimage captured through the first camera 2710 and the second imageobtained through the second camera 2720.

The electronic device 2300 may identify a visual object corresponding toa fixed external object of a plurality of visual objects included in theimage.

The electronic device 2300 may identify a first virtual plane spacedapart from the center point of the first lens of the first camera 2710by the focal length of the first lens in the first direction 2730. Forexample, the first virtual plane may be formed perpendicular to thefirst direction 2730. For example, the first virtual plane maycorrespond to the first image captured through the first camera 2710.

The electronic device 2300 may identify a second virtual plane spacedapart from the center point of the second lens of the second camera 2720by the focal length of the second lens in the first direction 2730. Forexample, the second virtual plane may be formed perpendicular to thefirst direction 2730. For example, the second virtual plane maycorrespond to the second image captured through the second camera 2720.

The electronic device 2300 may identify a first intersection pointbetween the first straight line extending from the center point of thefirst lens towards the fixed external object and the first virtualplane.

The electronic device 2300 may identify a second intersection pointbetween the second straight line extending from the center point of thesecond lens towards the fixed external object and the second virtualplane.

The electronic device 2300 may determine the geographic location of thefixed external object based on the first intersection point and thesecond intersection point.

In order to add a content corresponding to the fixed external object tothe electronic map, the electronic device 2300 may transmit informationon the geographic location of the fixed external object to the server2400. In other words, the information on the geographic location of thefixed external object may be transmitted to the server 2400 in order toadd the content corresponding to the fixed external object to theelectronic map.

Unlike the above-described embodiment, according to an embodiment, theelectronic device 2300 may identify a first angle θ1 between the firststraight line and a straight line perpendicular to the first direction2730. The electronic device 2300 may identify a second angle θ2 betweenthe second straight line and a straight line perpendicular to the firstdirection 2730. The electronic device 2300 may also obtain informationon the geographic location of the fixed external object, based on thedistance between the first camera 2710 and the second camera 2720, thefirst angle θ1, and the second angle θ2.

Hereinafter, a further detailed technical feature for determining thegeographic location of the fixed external object will be described withreference to FIG. 28.

Referring to 27 and 28, the electronic device 2300 may determine thegeographic location 2830 of the fixed external object, based on thefirst intersection point 2810 and the second intersection point 2820. Aprocess for determining the geographic location 2830 of the fixedexternal object will be described below.

The electronic device 2300 may identify the first virtual planes 2811spaced apart from the center point 2801 of the first lens 2815 of thefirst camera 2710 by the focal length 2812 of the first lens 2815 in thefirst direction 2730.

The electronic device 2300 may identify the first intersection point2810 between the first straight line extending from the center point2801 of the first lens 2815 towards the fixed external object and thefirst virtual plane 2811. The electronic device 2300 may identify athird intersection point 2813 between a third straight line extending inthe first direction 2730 from the center point 2801 of the first lens2815 and the first virtual plane 2811. The electronic device 2300 mayidentify a distance 2814 between the first intersection point 2810 andthe third intersection point 2813.

The electronic device 2300 may identify a second virtual plane 2821spaced apart from the center point 2802 of the second lens 2825 of thesecond camera 2720 by the focal length 2822 of the second lens 2825 inthe first direction 2730. The focal length 2822 of the second lens 2825may be set equal to the focal length 2812 of the first lens 2815.

The electronic device 2300 may identify the second intersection point2820 between the second straight line extending from the center point2802 of the second lens 2825 towards the fixed external object and thesecond virtual plane 2821. The electronic device 2300 may identify afourth intersection point 2823 between a fourth straight line extendingin the first direction 2730 from the center point 2802 of the secondlens 2825 and the second virtual plane 2821. The electronic device 2300may identify a distance 2824 between the second intersection point 2820and the fourth intersection point 2823.

The electronic device 2300 may identify a distance 2840 between thecenter point 2801 of the first lens and the center point 2802 of thesecond lens. The distance 2840 may correspond to a distance by which thesecond camera 2720 is separated from the first camera 2710.

The electronic device 2300 may determine the relative location of thefixed external object with respect to the center point 2801 of the firstlens 2815, based on the distance 2814, the distance 2824, and thedistance 2840.

A distance 2850 that is spaced apart from the center point 2801 of thefirst lens 2815 towards the fixed external object in the first direction2730 may be set as in Equation 5 below.

$\begin{matrix}{\mspace{79mu}{{Z = \frac{B \times F}{\left( \text{?} \right)}}{\text{?}\text{indicates text missing or illegible when filed}}}} & \left\lbrack {{Equation}\mspace{14mu} 5} \right\rbrack\end{matrix}$

Referring to the Equation 5, ‘Z’ is a distance 2850 by which thegeographical location 2830 of the fixed external object is spaced apartin the first direction 2730 with respect to the center point 2801 of thefirst lens 2815. ‘B’ is the distance 2840, ‘x_(r)’ is the distance 2814,and ‘x_(l)’ is the distance 2824. And, ‘F’ is the focal lengths 2812 and2822 of the first lens 2815 and the second lens 2825, respectively.

A distance 2860 by which the fixed external object is spaced apart inthe second direction 2740 perpendicular to the first direction 2730 withrespect to the center point 2801 of the first lens 2815 may be set as inEquation 6 below.

$\begin{matrix}{\mspace{79mu}{{X = \frac{B \times \text{?}}{\left( \text{?} \right)}}{\text{?}\text{indicates text missing or illegible when filed}}}} & \left\lbrack {{Equation}\mspace{14mu} 6} \right\rbrack\end{matrix}$

Referring to the Equation 6, ‘X’ is a distance 2850 by which thegeographical location 2830 of the fixed external object is spaced apartin the second direction 2740 perpendicular to the first direction 2730with respect to the center point 2801 of the first lens 2815. Further,‘B’ is the distance 2840, ‘x_(r)’ is the distance 2814, and ‘x_(l)’ isthe distance 2824.

Meanwhile, the electronic device 2300 may identify the geographiclocation of the electronic device 2300 (or the center point 2801 of thefirst lens 2815) through the GPS communication circuitry. For example,the electronic device 2300 may identify the latitude and longitude ofthe electronic device 2300 (or the center point 2801 of the first lens2815).

The electronic device 2300 may identify the geographic location of thefixed external object based on the geographic location of the electronicdevice 2300 (or the center point 2801 of the first lens 2815) and therelative location of the fixed external object with respect to theelectronic device 2300 (or the center point 2801 of the first lens2815).

According to an embodiment, at least a portion of the above-describedoperation of the electronic device 2300 may be performed by the server2400.

For example, the electronic device 2300 may transmit a first imageobtained through the first camera 2710 and a second image obtainedthrough the second camera 2720, to the server. The server 2400 mayreceive the first image and the second image from the electronic device2300.

The server 2400 may receive, from the electronic device 2300, additionalinformation (e.g., information on the geographic location of the secondlocation) for obtaining a geographic location of the fixed externalobject from the first image obtained through the first camera 2710 andthe second image obtained through the second camera 2720. The server2400 may obtain information on the geographic location of the fixedexternal object based on the additional information. For example, theserver 2400 may obtain information on the geographic location of thefixed external object, by performing the same or similar operation asthe above-described operation of the electronic device 2300.

FIG. 29 shows a view for explaining an embodiment for acquiringinformation on a fixed external object according to various embodimentsof the present disclosure;

Referring now to FIG. 29, the electronic device 2300 may obtain an imageincluding a plurality of visual objects respectively corresponding to aplurality of external objects, through a camera 2330.

The electronic device 2300 may analyze the obtained image. Theelectronic device 2300 may identify a visual object corresponding to thefixed external object 2910 among the plurality of visual objects, basedon a result of the analysis of the obtained image. For example, theelectronic device 2300 may identify a plurality of visual objectsrespectively corresponding to the plurality of external objects includedin the obtained image. The electronic device 2300 may classify theplurality of visual objects into one of a dynamic object and a staticobject, respectively. The electronic device 2300 may determine whetherthe plurality of external objects respectively corresponding to theplurality of visual objects are stationary or movable. Accordingly, theelectronic device 2300 may identify the visual object corresponding tothe fixed external object 2910.

After identifying the visual object corresponding to the fixed externalobject 2910, the electronic device 2300 may obtain information on thevisual object. The information on the visual object may includeinformation on the fixed external object 2910.

As an example, the electronic device 2300 may identify a tree-shapedvisual object and then identify that a fixed external object 2910corresponding to the visual object is a roadside tree. That is, theelectronic device 2300 may obtain information on the fixed externalobject 2910 being a roadside tree.

As another example, the electronic device 2300 may identify a roadsign-shaped visual object and then identify that a fixed external object2910 corresponding to the visual object is a road sign. Afteridentifying the fixed external object 2910 as a sign, the electronicdevice 2300 may obtain information on the content of the sign, based onan optical character recognition (OCR) process.

As another example, the electronic device 2300 may identify abuilding-shaped visual object and then identify that a fixed externalobject 2910 corresponding to the visual object is a building. Afteridentifying the fixed external object 2910 as a building, the electronicdevice 2300 may obtain information on a store or stores located withinthe building, based on the optical character recognition process for asignboard of the building.

As another example, the electronic device 2300 may identify a visualobject representing a traffic accident scene and then identify thetraffic accident scene. The electronic device 2300 may obtaininformation that a traffic accident has occurred, based on theidentifying the visual object.

As another example, the electronic device 2300 may identify a visualobject representing some crack (or a sinkhole, etc.) on a road and thenidentify that the crack (or the sinkhole, etc.) exists is on the road.The electronic device 2300 may obtain information that such a crack (ora sinkhole, etc.) exists on the road, based on identifying the visualobject.

According to an embodiment, the operation of analyzing the image asdescribed above may be performed in the server 2400. The electronicdevice 2300 may transmit the obtained image to the server 2400. Theserver 2400 may identify the visual object corresponding to the fixedexternal object 2910 of a plurality of visual objects, based onanalyzing the image received from the electronic device 2300.

In the meantime, the electronic device 2300 may obtain information on adistance between the fixed external object 2910 and the electronicdevice 2300, obtained through the sensing circuit 2340. For example, thesensing circuit 2340 may include a transmitter configured to emit orirradiate a signal and a receiver configured to receive a reflectedsignal for the signal. The electronic device 2300 may obtain informationon the distance between the fixed external object 2910 and theelectronic device 2300, by means of emitting a signal through thesensing circuit 2340 and then receiving its reflected signal.

According to an embodiment, the information on the distance between thefixed external object 2910 and the electronic device 2300 may bedistinguished from information on the visual object. For example, theinformation on the visual object may include data on the shape of thefixed external object 2910, data on the type of the fixed externalobject 2910, data on the function of the fixed external object 2910,and/or the like. For example, the information on the distance betweenthe fixed external object 2910 and the electronic device 2300 mayinclude information on the distance and the direction (or angle) bywhich the electronic device 2300 is space apart from the fixed externalobject 2910. In other words, the electronic device 2300 may obtain theinformation on the visual object through the camera 2330, and may obtainthe information between the distance between the fixed external object2910 and the electronic device 2300, through the sensing circuit 2340.

According to an embodiment, the electronic device 2300 may identifyinformation on the fixed external object 2910 through the sensingcircuit 2340 other than the camera 2330. The electronic device 2300 mayobtain the information on the distance between the fixed external object2910 and the electronic device 2300 through the sensing circuit 2340, aswell as the information on the type of the fixed external object 2910,the shape of the fixed external object 2910 and so on.

Meanwhile, the electronic device 2300 may transmit to the server 2400the information on the distance between the fixed external object 2910and the electronic device 2300 and the information on the visual object,obtained through the sensing circuit 2340. The information on thedistance between the fixed external object 2910 and the electronicdevice and the information on the visual object may be transmitted tothe server for adding the content corresponding to the fixed externalobject 2910 to an electronic map.

According to embodiments, the electronic device 2300 may identify aplurality of external objects prior to the camera 2330, through thesensing circuit 2340. The electronic device 2300 may identify existenceof a plurality of external objects through the sensing circuit 2340,before acquiring an image through the camera 2330. The electronic device2300 may capture the plurality of external objects through the camera2330 and obtain an image including a plurality of visual objects eachcorresponding to the plurality of external objects. Based on the imageobtained by the electronic device 2300, the information on the visualobject may be obtained by identifying a visual object corresponding tothe fixed external object 2910.

FIGS. 30 to 32 illustrate examples of operations of an electronic deviceaccording to various embodiments of the present disclosure.

Referring now to FIG. 30, the electronic device 2300 within a movingvehicle may identify a fixed external object 3003 outside the electronicdevice 2300 (or the vehicle) through the camera 2330. For example, theelectronic device 2300 may obtain an image corresponding to the outsideof the electronic device 2300 through the camera 2330. The image mayinclude a plurality of visual objects each corresponding to a pluralityof external objects. The electronic device 2300 may analyze the obtainedimage. The electronic device 2300 may determine whether the plurality ofvisual objects are dynamic objects or static objects, based on analyzingthe obtained image. For example, the electronic device 2300 may obtain aplurality of images and then perform image registration of the pluralityof images, so as to determine whether the plurality of visual objectsare dynamic objects or static objects, by according to theabove-described embodiment.

For example, the electronic device 2300 may determine the externalobject 3001 and the external object 3002 as a movable external object.The electronic device 2300 may determine the external object 3003 as afixed external object.

Referring then to FIGS. 30 and 31, the electronic device 2300 mayidentify that a content representing the external object 3003 is notincluded in the electronic map 3100 displayed through an applicationexecuted in the electronic device 2300. For example, the electronicdevice 2300 may identify the content for a location 3110 on theelectronic map 3100, corresponding to the location of the externalobject 3003. The electronic device 2300 may identify that the contentrepresenting the external object 3003 is different from the content forthe location 3110 on the electronic map 3100 corresponding to thelocation of the external object 3003, or it is not displayed on thelocation 3110 in the electronic map 3100 corresponding to the locationof the external object 3003. In addition, the electronic device 2300 maydisplay the content 3120 for indicating the location of the electronicdevice 2300 on the electronic map 3100.

The electronic device 2300 may obtain information on a visual object byidentifying the visual object corresponding to the external object 3003.The electronic device 2300 may obtain information on a distance betweenthe external object 3003 and the electronic device 2300 through thesensing circuit 2340. The electronic device 2300 may transmitinformation on the visual object and information on the distance betweenthe external object 3003 and the electronic device 2300, to the server2400. The server 2400 may receive information on the visual object andthe information on the distance between the external object 3003 and theelectronic device 2300. The server 2400 may add the contentcorresponding to the external object 3003 to the electronic map, basedon the information on the visual object and the information on thedistance between the external object 3003 and the electronic device2300. The server 2400 may transmit, to the electronic device 2300, theinformation on the electronic map added with the content.

Referring now to FIGS. 30 to 32, the electronic device 2300 may receivethe information on the electronic map 3200 to which the content 3210corresponding to the external object 3003 is added.

For example, the electronic device 2300 may display the receivedelectronic map 3100 on the display 2360. The electronic device 2300 maytransmit to the server 2400 the information on the external object 3003and the information on the distance between the external object 3003 andthe electronic device 2300, based on that the content 3210 representingthe external object 3003 is not displayed on the electronic map 3100.

The electronic device 2300 may receive from the server 2400 theinformation on the electronic map 3200 including the content 3210representing the external object 3003, and display the receivedelectronic map 3200 on the display 2360.

In other words, the electronic device 2300 may identify the externalobject 3003 that is not displayed on the electronic map 3100 and add thecontent representing the external object 3003 to the electronic map 3100in real time, so as to update the electronic map 3100. In addition, theelectronic device 2300 may display a content 3220 for indicating thelocation of the electronic device 2300 through the updated electronicmap 3200.

According to the foregoing embodiments, it has been described that theoperation of identifying the information on the fixed external objectand the information on the geographic location of the fixed externalobject is performed in the electronic device 2300, but it is only forconvenience of description. Then, the operation of identifying theinformation on the fixed external object and the information on thegeographic location of the fixed external object may be performed by theserver 2400.

FIGS. 33 and 34 each show an example of operations of a server accordingto various embodiments.

Referring now to FIG. 33, the server 2400 may store an electronic mapand transmit the stored electronic map to a vehicle (or an electronicdevice). In the electronic map, as geographic information on roads,buildings, and/or bridges requires frequent and continuous updates, theserver 2400 may be adapted to receive information on certain externalobjects through a plurality of vehicles (or a plurality of electronicdevices), and update the electronic map based on the information onthose external objects.

The server 2400 may obtain the information on a fixed external object3310, based on information received from the first electronic device3301, the second electronic device 3302, and the third electronic device3303. As the first electronic device 3301, the second electronic device3302 and the third electronic device 3303 are moving in differentdirections from different positions from each other, the firstelectronic device 3301, the second electronic device 3302 and the thirdelectronic device 3303 may transmit different information to the server2400.

For example, the first electronic device 3301 may identify a firstvisual object corresponding to the fixed external object 3310. The firstelectronic device 3301 may transmit, to the server 2400, information ona distance between the fixed external object 3310 and the firstelectronic device 3301, and information on the first visual object. Theserver 2400 may receive the information on the distance between thefixed external object 3310 and the first electronic device 3301 and theinformation on the first visual object.

The server 2400 may obtain first information on the fixed externalobject 3310 and information on a first geographic location of the fixedexternal object 3310, based on the information on the distance betweenthe fixed external object 3310 and the first electronic device 3301, andthe information on the first visual object.

For example, the second electronic device 3302 may identify a secondvisual object corresponding to the fixed external object 3310. Thesecond electronic device 3302 may transmit, to the server 2400,information on a distance between the fixed external object 3310 and thesecond electronic device 3302 and information on the second visualobject. The server 2400 may receive the information on the distancebetween the fixed external object 3310 and the second electronic device3302 and the information on the second visual object.

The server 2400 may obtain second information on the fixed externalobject 3310 and information on a second geographic location of the fixedexternal object 3310, based on the information on the distance betweenthe fixed external object 3310 and the second electronic device 3302,and the information on the second visual object.

For example, the third electronic device 3303 may identify a thirdvisual object corresponding to the fixed external object 3310. The thirdelectronic device 3303 may transmit, to the server 2400, information ona distance between the fixed external object 3310 and the thirdelectronic device 3303, and information on the third visual object. Theserver 2400 may receive the information on the distance between thefixed external object 3310 and the third electronic device 3303, and theinformation on the third visual object.

The server 2400 provides information on the fixed external object 3310based on the information on the distance between the fixed externalobject 3310 and the third electronic device 3303 and the information onthe third visual object. The third information and information on thethird geographic location of the fixed external object 3310 may beobtained.

The first to third information is of information each obtained throughthe first electronic device 3301 to the third electronic device 3303,and may be different from each other. In other words, the first to thirdinformation may include some errors or tolerance. The server 2400 mayobtain accurate information on the fixed external object 3310 based onthe first to third information. For example, the server 2400 maycalibrate the information on the fixed external object 3310, based onthe first to third information, to obtain accurate information on thefixed external object 3310.

For example, the server 2400 may identify the fixed external object 3310as a traffic light, based on the first information. The server 2400 mayidentify the fixed external object 3310 as a traffic sign, based on thesecond information. The server 2400 may identify the fixed externalobject 3310 as a traffic light, based on the third information. Theserver 2400 may then identify the fixed external object 3310 as atraffic light, based on the first to third information.

In the meantime, the information on the first to third geographiclocations may be different from each other, as they are obtained basedon information obtained through the first to third electronic devices3301 to 3303. In other words, the information on the first to thirdgeographic locations may include some errors or tolerance. The server2400 may obtain accurate information on the geographic location of thefixed external object 3310, based on the information on the first tothird geographic locations. For example, the server 2400 may calibratethe information on the geographic location of the fixed external object3310, based on the information on the first to third geographiclocations, so as to obtain more accurate information on the geographiclocation of the fixed external object 3310.

Referring to FIGS. 33 and 34, the server 2400 may update the electronicmap 3400, based on the accurate information on the fixed external object3310 and the accurate information on the geographic location of thefixed external object 3310. For example, the server 2400 may add acontent 3410 representing the fixed external object 3310 to a locationcorresponding to the geographic location of the fixed external object3310 on the electronic map 3400.

Further, the server 2400 may transmit information on the updatedelectronic map 3400 to the first to third electronic devices 3301 to3303. For example, the first to third electronic devices 3301 to 3303may receive the information on the updated electronic map 3400. Forexample, the first electronic device 3301 may display the updatedelectronic map 3400 on a display of the first electronic device 3301.

According to embodiments, the electronic device (e.g., the firstelectronic device 3301) may perform at least some of the functions ofthe aforementioned server 2400.

For example, the first electronic device may perform at least some ofthe functions of the above-described server 2400. The first electronicdevice 3301 may receive, from the second electronic device 3302,information on a distance between the fixed external object 3310 and thesecond electronic device 3302, and information on the second visualobject. The first electronic device 3301 may obtain second informationon the fixed external object 3310 and the information on the secondgeographic location of the fixed external object 3310, based on theinformation on the distance between the fixed external object 3310 andthe second electronic device 3302, and the information on the secondvisual object.

The first electronic device 3301 may receive, from the third electronicdevice 3303, information on a distance between the fixed external object3310 and the third electronic device 3303, and information on the thirdvisual object. The first electronic device 3301 may obtain the thirdinformation on the fixed external object 3310 and the information on thethird geographic location of the fixed external object 3310, based onthe information on the distance between the fixed external object 3310and the third electronic device 3303 and the information on the thirdvisual object.

The first electronic device 3301 may obtain accurate information on thefixed external object 3310, based on the second information and thethird information. The first electronic device 3301 may obtain accurateinformation on the geographic location of the fixed external object3310, based on the information on the second geographic location and theinformation on the third location.

The first electronic device 3301 may update the electronic map. Thefirst electronic device 3301 may transmit information on the updatedelectronic map to the second electronic device 3302 and the thirdelectronic device 3303.

FIG. 35 shows an example for identifying a geographic location of afixed external object according to various embodiments.

Referring to FIG. 35, the electronic device 2300 may be incorporatedinto a moving vehicle. The electronic device 2300 may identify in whichlane the vehicle is driving. For example, the electronic device 2300 mayidentify in which lane the vehicle is driving, based on informationobtained through the GPS communication circuitry or the camera 2330.

The electronic device 2300 may obtain information on a width of the lanein which the vehicle is traveling. For example, the electronic device2300 may receive information on the width of the lane in which thevehicle is traveling, through the server 2400. As another example, theelectronic device 2300 may obtain information on the width of the lanein which the vehicle is traveling, based on information on the width ofthe lane included in the electronic map.

The electronic device 2300 may identify a fixed external object 3510.The electronic device 2300 may identify a distance between the fixedexternal object 3510 and the electronic device 2300. The electronicdevice 2300 may identify an angle between a first straight lineextending from the location of the electronic device 2300 to thelocation of the fixed external object 3510 and a second straight lineperpendicular to the driving lane. The electronic device 2300 mayidentify the geographical location of the fixed external object 3510,based on the angle between the first straight line and the secondstraight line perpendicular to the driving lane, and the width of thelane.

For example, the fixed external object 3510 may be located at one sideend of the road. The electronic device 2300 may identify the relativelocation of the fixed external object 3510 with respect to theelectronic device 2300, in order to identify the geographic location ofthe fixed external object 3510. When the vehicle with the electronicdevice 2300 is driving in a second lane of three lanes of the road, theelectronic device 2300 may display the relative location of the fixedexternal object 3510, on the basis of the electronic device 2300, usingthe Equations 7 and 8 below.

$\begin{matrix}{X = \frac{2a}{2}} & \left\lbrack {{Equation}\mspace{14mu} 7} \right\rbrack\end{matrix}$

Referring to the Equation 7, ‘X’ is a distance by which the geographiclocation 2630 of the fixed external object is spaced apart in the seconddirection perpendicular to the first direction with respect to theelectronic device 2300, and ‘a’ is the width of the lane.

$\begin{matrix}{Z = {\frac{2a}{2}\tan\;\theta}} & \left\lbrack {{equation}\mspace{14mu} 8} \right\rbrack\end{matrix}$

Referring to the Equation 8, ‘Z’ is a distance by which the geographicallocation 2630 of the fixed external object is space apart in thetraveling direction of the vehicle with respect to the electronic device2300, and ‘a’ is the width of the lane. Further, ‘0’ is the anglebetween the first straight line extending from the location of theelectronic device 2300 to the location of the fixed external object 3510and the second straight line perpendicular to the lane of the road.

The electronic device 2300 may identify the relative location of thefixed external object 3510 with respect to the electronic device 2300,based on the Equations 7 and 8. The electronic device 2300 may identifythe current location of the vehicle based on the GPS communicationcircuitry. The electronic device 2300 may identify the geographiclocation of the fixed external object 3510, based on the current vehiclelocation and the relative location of the fixed external object 3510.

According to embodiments, the electronic device 2300 may identify thedistance between the fixed external object 3510 and the electronicdevice 2300. The electronic device 2300 may identify the angle betweenthe first straight line extending from the location of the electronicdevice 2300 to the location of the fixed external object 3510 and thesecond straight line perpendicular to the lane of the road. Theelectronic device 2300 may identify the geographic location of the fixedexternal object 3510, based on the angle between the first straight lineand the second straight line, and the distance between the fixedexternal object 3510 and the electronic device 2300.

X=b cos θ  [Equation 9]

Referring to the Equation 9, ‘X’ is a distance by which the geographicallocation 2630 of the fixed external object is spaced apart from theelectronic device 2300 in the second direction perpendicular to thefirst direction. ‘b’ is the distance between the fixed external object3510 and the electronic device 2300. ‘θ’ is the angle between the firststraight line extending from the location of the electronic device 2300to the location of the fixed external object 3510 and the secondstraight line perpendicular to the lane of the road.

Z=b sin θ  [Equation 10]

Referring to the Equation 10, ‘Z’ is the distance by which thegeographical location 2630 of the fixed external object is spaced apartin the traveling direction of the vehicle from the electronic device2300. ‘b’ is the distance between the fixed external object 3510 and theelectronic device 2300), and ‘θ’ is the angle between the first straightline extending from the location of the electronic device 2300 to thelocation of the fixed external object 3510 and the second straight lineperpendicular to the lane of the road.

According to embodiments, the electronic device 2300 may identify thedistance between the external object 3510 and the electronic device2300, based on the information on the width of the lane in which thevehicle with the electronic device 2300 is traveling, the focal length,and a width of a visual object corresponding to the lane of the roadwithin the image obtained through the camera 2330 of the electronicdevice 2300. The electronic device 2300 may identify the geographiclocation of the external object 3510, based on the distance between theexternal object 3510 and the electronic device 2300.

According to embodiments, the electronic device 2300 may transmit, tothe server 2400, the information on the location of the electronicdevice 2300 and the information on the angle between the first straightline extending from the location of the electronic device 2300 to thelocation of the fixed external object 3510 and the second straight lineperpendicular to the lane of the road. The server 2400 may obtaininformation on the width of the lane in which the vehicle with theelectronic device 2300 is traveling. The server 2400 may identify thegeographic location of the fixed external object 3510, based on theinformation on the location of the electronic device 2300, theinformation on the angle between the first straight line and the secondstraight line, and the information on the width of the lane of road.

According to embodiments, the electronic device 2300 may transmit, tothe server 2400, the information on the location of the electronicdevice 2300, the information on the angle between the first straightline extending from the location of the electronic device 2300 to thelocation of the fixed external object 3510 and the second straight lineperpendicular to the lane of the road, and the information on thedistance between the external object 3510 and the electronic device2300. The server 2400 may obtain information on the width of the lane inwhich the vehicle including the electronic device 2300 is traveling. Theserver 2400 may identify the geographic location of the fixed externalobject 3510, based on the information on the location of the electronicdevice 2300, the information on the angle between the first straightline and the second straight line, and the information on the distancebetween the external object 3510 and the electronic device 2300.

FIG. 36 shows an example for identifying a geographic location of afixed external object according to various embodiments of the presentdisclosure.

Referring to FIG. 36, the server 2400 may receive information on thefirst geographic location of the fixed external object 3610 from thefirst electronic device 3601. The server 2400 may receive information onthe second geographic location of the fixed external object 3610 fromthe second electronic device 3602. The server 2400 may identifyinformation on the accurate geographic location of the fixed externalobject 3610, based on the information on the first geographic locationand the information on the second geographic location. For example, theserver 2400 may identify the information on the accurate geographiclocation of the fixed external object 3610, by calculating the averageof the first geographic location and the second geographic location.

According to an embodiment, the server 2400 may receive a first imageincluding a first visual object corresponding to the fixed externalobject 3610 from the first electronic device 3601. The server 2400 mayreceive a second image including a second visual object corresponding tothe fixed external object 3610 from the second electronic device 3602.The server 2400 may identify (or estimate) a first geographic locationof the fixed external object 3610 based on the first image. The server2400 may identify (or estimate) a second geographic location of thefixed external object 3610 based on the second image. The server 2400may identify information on the accurate geographic location of thefixed external object 3610, by calculating the average of the firstgeographic location and the second geographic location.

FIG. 37 shows an example of an operation of an electronic deviceaccording to various embodiments of the present disclosure.

Referring to FIG. 37, the electronic device may be related to orassociated with the electronic device 2300 of FIG. 23. For example, theelectronic device may be incorporated into in a moving vehicle.

In operation 3701, the electronic device (e.g., the processor of theelectronic device) may obtain an image including a plurality of visualobjects respectively corresponding to a plurality of external objectsthrough the camera. The plurality of visual objects may include a visualobject corresponding to a fixed external object.

For example, the camera may be located in front of the vehicle. Theelectronic device may obtain an image including a plurality of visualobjects respectively corresponding to a plurality of external objectslocated in front of the vehicle, through the camera.

According to an embodiment, the electronic device may identify whether afirst content corresponding to the fixed external object exists in alocation on the electronic map corresponding to the geographic locationof the fixed external object. The electronic device may obtain the imageincluding the plurality of visual objects respectively corresponding tothe plurality of external objects, based on whether the first contentexists or not.

For example, the electronic device may obtain the image including theplurality of visual objects respectively corresponding to the pluralityof external objects, based on the absence of the first content at thelocation on the electronic map corresponding to the geographicallocation of the fixed external object.

According to an embodiment, the electronic device may identify whetherthe second content is related to the fixed external object, at thelocation on the electronic map corresponding to the location of thefixed external object. The electronic device may obtain the imageincluding a plurality of visual objects, based on whether or not thesecond content is related to the fixed external object.

For example, the electronic device may identify that the second contentis displayed at a location on the electronic map corresponding to thelocation of the fixed external object. The electronic device may obtainthe image including a plurality of visual objects, based on determiningthat the second content is not related to the fixed external object. Forexample, when the fixed external object is a roadside tree, theelectronic device may identify that the content representing a buildingis displayed at the location on the electronic map corresponding to thelocation of the roadside tree. Since the content representing thebuilding is not associated with to the roadside tree, the electronicdevice may obtain an image including a plurality of visual objects.

In operation 3702, the electronic device may obtain information on thevisual object, by identifying the visual object corresponding to thefixed external object among a plurality of visual objects, based on aresult of analysis of the obtained image.

For example, the electronic device may classify the plurality of visualobjects into one of a dynamic object and a static object, based on theresult of analysis of the obtained image. For example, the visual objectcorresponding to the fixed external object may be classified as a staticobject.

For example, the information on the visual object may include data aboutthe shape of the fixed external object, data about the type of the fixedexternal object, and data about the function of the fixed externalobject.

According to an embodiment, the electronic device may analyze theobtained image. The electronic device may identify the visual objectcorresponding to the fixed external object among the plurality of visualobjects. The electronic device may obtain information on the visualobject, by identifying the visual object corresponding to the fixedexternal object.

For example, the electronic device may obtain data on the type of thefixed external object based on the visual object corresponding to thefixed external object among the plurality of visual objects. As anexample, the electronic device may obtain information on the width ofthe fixed external object. The electronic device may obtain data on thetype of the fixed external object through the machine learning model,based on the information on the width of the fixed external object.According to an embodiment, the electronic device may transmit theinformation on the width of the fixed external object to the server forperforming a machine learning model, and receive data on the type of thefixed external object through the server for performing the machinelearning model.

According to an embodiment, the electronic device may transmit theobtained image to the server. The electronic device may receiveinformation on the result of analysis of the obtained image from theserver.

For example, the obtained image may be used together with a referenceimage to extract a plurality of feature points. In other words, theserver may extract the plurality of feature points based on the obtainedimage and the reference image. The plurality of feature points, whichare points at which an image brightness rapidly changes in the referenceimage or the obtained image, may include an edge of pixel or a cornerpoint. The plurality of feature points may include first feature pointsextracted from the reference image and second feature points extractedfrom the obtained image.

According to an embodiment, the electronic device may transmit, to theserver, information on the angle at which the obtained image wascaptured and the direction in which the obtained image was captured,together with the obtained image.

In operation 3703, the electronic device may transmit, to the server,information on a distance between a fixed external object and theelectronic device, and information on a visual object, obtained throughthe sensing circuit. The information on the distance between the fixedexternal object and the electronic device, and the information on thevisual object may be transmitted to the server to add the first contentcorresponding to the fixed external object to the electronic map.

The electronic device may receive information for displaying the firstcontent on the electronic map within the application for displaying theelectronic map. The electronic device may display the electronic map towhich the first content is added, through the display.

According to an embodiment, the electronic device may obtain informationon the distance between the fixed external object and the electronicdevice through a sensing circuit. For example, the sensing circuit mayinclude a transmitter configured to emit a signal and a receiverconfigured to receive its reflected signal. For example, the sensingcircuit may include a radar sensor or a lidar sensor.

According to an embodiment, the electronic device may identify angleinformation between a straight line perpendicular to the travellingdirection of the vehicle and another straight line extending from thelocation of the vehicle (or the location of the electronic device)towards the fixed external object. The electronic device may transmit tothe server the angle information together with the information on thedistance between the fixed external object and the electronic device.

According to an embodiment, the electronic device may also obtain theinformation on the distance between the fixed external object and theelectronic device through the camera. The electronic device may obtainthe information on the distance between the fixed external object andthe electronic device, based on the width of the visual object and thefocal length of the camera. For example, the electronic device mayobtain the information on the width of the fixed external object. Theinformation on the width of the fixed external object may includeinformation on the average width of the fixed external object. Theelectronic device may obtain the information on the distance between thefixed external object and the electronic device, based on the averagewidth of the fixed external object, the width of the visual object, andthe focal length of the camera. For example, the electronic device mayobtain the information on the distance between the fixed external objectand the electronic device using the Equation 11 below.

D=W×(f+w)  [Equation 11]

Referring to the Equation 11, ‘D’ is the distance between the fixedexternal object and the electronic device. ‘W’ is the average width ofthe fixed external object, ‘f’ is the focal length of the camera, and‘w’ is the width of the visual object.

According to an embodiment, the electronic device may obtain theinformation on the geographic location of the electronic device throughthe GPS communication circuitry. The electronic device may identify theinformation on the geographic location of the fixed external object,based on the information on the geographic location of the electronicdevice and the information on the distance between the fixed externalobject and the electronic device. The electronic device may transmit theinformation on the geographic location of the fixed external object tothe server.

According to an embodiment, the electronic device may obtain theinformation on the width of a lane in which a moving vehicle is located.The electronic device may obtain the information on the width of thefixed external object, based on the information on the width of the laneof the road. The electronic device may transmit the information on thefixed width of the external object to the server. The information on thewidth of the fixed external object may be used to obtain data about thetype of the fixed external object.

According to an embodiment, the electronic device may identify that thefixed external object is located within a predetermined radius of theelectronic device. The electronic device may establish a connection withthe fixed external object. The electronic device may receive theinformation on the geographic location of the fixed external object andthe information on the fixed external object, from the fixed externalobject. The electronic device may transmit the information on thegeographic location of the fixed external object to the server.

According to embodiments, the electronic device may identify that anexternal electronic device is located within a predetermined radius ofthe electronic device. The electronic device may establish a connectionwith the external electronic device, based on identifying that theexternal electronic device is located within the predetermined radius ofthe electronic device. The electronic device may receive the informationon the distance between the fixed external object and the externalelectronic device, from the external electronic device. The electronicdevice may obtain the information on the geographic location of thefixed external object, based on the information on the distance betweenthe fixed external object and the electronic device, and the informationon the distance between the fixed external object and the externalelectronic device. The electronic device may transmit the information onthe geographic location of the fixed external object to the server.

FIG. 38 shows another example of an operation of an electronic deviceaccording to various embodiments of the present disclosure.

Referring to FIG. 38, the electronic device may be related to theelectronic device 2300 of FIG. 23. For example, the electronic devicemay be incorporated into a moving vehicle.

In operation 3801, the electronic device (e.g., a processor of theelectronic device) may obtain a first image corresponding to theexterior of the vehicle through a camera, at a first location, while theelectronic device in the vehicle is moving in a first direction.

For example, the electronic device may obtain the first imagecorresponding to the outside of the vehicle, through a camera, at afirst location capable of identifying a fixed external object, based onthat a first content corresponding to a fixed external object is notdisplayed on the electronic map.

In operation 3802, the electronic device may identify a first visualobject corresponding to the fixed external object among a plurality ofvisual objects included in the first image. For example, the electronicdevice may identify the first visual object based on a result ofanalysis of the first image. The electronic device may classify theplurality of visual objects into one of a static object and a dynamicobject, and may identify the first visual object classified as thestatic object.

In operation 3803, while the electronic device is located in the firstlocation, the electronic device may identify a first virtual planespaced apart from the center point of lens of the camera by a focallength of the lens in a first direction. For example, the first virtualplane may correspond to the first image.

In operation 3804, at the first location, the electronic device mayidentify a first intersection point between a first straight lineextending from the center point of the lens towards the fixed externalobject and the first virtual plane.

In operation 3805, after moving from the first location in the firstdirection, the electronic device may obtain a second image correspondingto the exterior of the vehicle at a second location through the camera.

In operation 3806, the electronic device may identify a second visualobject corresponding to the fixed external object among a plurality ofvisual objects included in the second image. For example, the firstvisual object and the second visual object may correspond to the fixedexternal object. The first visual object and the second visual objectare objects included in images obtained from different locations, andmay have different dimensions or shapes.

In operation 3807, the electronic device may identify a second virtualplane spaced apart from the center point of the lens of the camera bythe focal length of the lens in the first direction, while theelectronic device is located in the second location. For example, thesecond virtual plane may correspond to the second image.

In operation 3808, at the second location, the electronic device mayidentify a second intersection point between a second straight lineextending from the center point of the lens towards the fixed externalobject and the second virtual plane.

In operation 3809, the electronic device may determine the geographiclocation of the fixed external object based on the first intersectionpoint and the second intersection point.

According to an embodiment, the electronic device may identify the firstlocation and the second location through a GPS communication circuitry.The electronic device may determine the relative location of the fixedexternal object with respect to the second location, based on the firstintersection point and the second intersection point. The electronicdevice may determine the geographic location of the fixed externalobject based on the determined relative location.

According to an embodiment, the electronic device may identify speedinformation of the electronic device between the first location and thesecond location through the speedometer. The electronic device mayidentify time information required to move from the first location tothe second location. The electronic device may obtain a distance betweenthe first location and the second location based on the speedinformation and the time information. The electronic device maydetermine the geographic location of the fixed external object based onthe obtained distance between the first location and the secondlocation. For example, the electronic device may determine the relativelocation of the fixed external object with respect to the secondlocation, based on the obtained distance between the first location andthe second location, and the first intersection point and the secondintersection point. The electronic device may determine the geographiclocation of the fixed external object based on the geographic locationof the second location.

According to an embodiment, the electronic device may obtain informationon the width of the lane in which the vehicle is travelling. Theelectronic device may identify an angle between the first direction andthe first straight line. The electronic device may determine thegeographic location of the fixed external object, based on the width ofthe lane of the road and the angle. For example, the electronic devicemay determine the relative location of the fixed external object on thebasis of the first location. The electronic device may determine thegeographic location of the fixed external object based on the geographiclocation of the first location.

In operation 3810, the electronic device may transmit the information onthe geographic location of the fixed external object to the server. Forexample, the information on the geographic location of the fixedexternal object may be transmitted to the server to add the contentcorresponding to the fixed external object onto the electronic map.

According to an embodiment, the electronic device may obtain data on thetype of the fixed external object based on the first visual object andthe second visual object. The electronic device may transmit the data onthe type of the fixed external object to the server together withinformation on the geographic location of the fixed external object.

FIG. 39 shows another example of an operation of an electronic deviceaccording to various embodiments of the present disclosure.

Referring now to FIG. 39, the electronic device may be related to theelectronic device 2300 of FIG. 23. For example, the electronic devicemay be incorporated into in a moving vehicle. The electronic device mayinclude two or more cameras. As an example, the electronic device mayinclude a first camera and a second camera.

In operation 3901, the electronic device may obtain an imagecorresponding to the exterior of the vehicle through at least one of thefirst camera and the second camera.

For example, the electronic device may obtain the image corresponding tothe exterior of the vehicle, based on identifying that the first contentcorresponding to the fixed external object is not displayed on theelectronic map.

In operation 3902, the electronic device may identify a visual objectcorresponding to the fixed external object of a plurality of visualobjects included in the image. For example, the electronic device mayidentify the visual object corresponding to the fixed external objectbased on a result of analysis of the image. The electronic device mayclassify the plurality of visual objects into one of a static object anda dynamic object, and may identify the visual object corresponding tothe fixed external object classified as the static object.

In operation 3903, the electronic device may identify a first virtualplane spaced apart from the center point of the first lens of the firstcamera by the focal length of the first lens in the first direction.

In operation 3904, the electronic device may identify a second virtualplane spaced apart from the center point of the second lens of thesecond camera by the focal length of the second lens in the firstdirection.

In operation 3905, the electronic device may identify a firstintersection point between a first straight line extending from thecenter point of the first lens towards the fixed external object and thefirst virtual plane.

In operation 3906, the electronic device may identify a secondintersection point between a second straight line extending from thecenter point of the second lens towards the fixed external object, andthe second virtual plane.

In operation 3907, the electronic device may determine the geographiclocation of the fixed external object based on the first intersectionpoint and the second intersection point. For example, the electronicdevice may determine the relative location of the fixed external objectwith respect to the location of the vehicle, based on the firstintersection point and the second intersection point. The electronicdevice may obtain the location of the vehicle through the GPScommunication circuitry. The electronic device may determine thegeographic location of the fixed external object, based on the locationof the vehicle and the relative location of the fixed external object.

In operation 3908, the electronic device may transmit the information onthe geographic location of the fixed external object to the server. Forexample, the information on the geographic location of the fixedexternal object may be sent to a server to add the content correspondingto the fixed external object onto the electronic map.

FIG. 40 shows an example of an operation of the first electronic deviceaccording to various embodiments of the present disclosure.

Referring now to FIG. 40, the first electronic device may be related tothe server 2400 of FIG. 24.

In operation 4001, the first electronic device may obtain, from thesecond electronic device, a first image including a first visual objectcorresponding to the fixed external object.

In operation 4002, the first electronic device may obtain, from thethird electronic device, a second image including a second visual objectcorresponding to the fixed external object.

In operation 4003, the first electronic device may obtain information onthe geographic location of the fixed external object, based on the firstimage and the second image.

According to an embodiment, the first electronic device may classify aplurality of visual objects included in the first image and the secondimage into one of a static object and a dynamic object, respectively,based on the first image and the second image. The first electronicdevice may identify a first visual object corresponding to the fixedexternal object of at least one visual object classified as the staticobject.

The first electronic device may estimate (or identify) the type of thefirst visual object based on e.g., a machine learning model. The firstelectronic device may learn the type of the first visual object based onthe machine learning model. The training of the machine learning modelmay include adjusting weights between a plurality of nodes included inneural networks (such as, for example, a feedforward neural network, aconvolution neural network (CNN), a recurrent neural network (RNN),and/or a long-short term memory Model (LSTM)), based on supervisedlearning basics and/or unsupervised learning basics.

According to an embodiment, the first electronic device may receive,from the second electronic device, additional information for obtainingthe information on the first geographic location of the fixed externalobject through the first image. The first electronic device may obtainthe information on the first geographic location of the fixed externalobject, by analyzing the first image based on the additionalinformation. For example, the first electronic device may receive, fromthe second electronic device, information on the focal length of thecamera of the second electronic device and information on a geographiclocation at which the first image is captured.

The first electronic device may receive, from the third electronicdevice, additional information for obtaining information on the secondgeographic location of the fixed external object through the secondimage. The first electronic device may obtain information on the secondgeographic location of the fixed external object, by analyzing thesecond image based on the additional information.

The first electronic device may obtain the information on the geographiclocation of the fixed external object, based on the information on thefirst geographic location and the information on the second geographiclocation. The information on the geographic location of the fixedexternal object may be obtained based on the average of the informationon the first geographic location and the information on the secondgeographic location.

According to an embodiment, the first electronic device may receiveinformation on the first geographic location of the fixed externalobject from the second electronic device. The first electronic devicemay receive information on the second geographic location of the fixedexternal object from the third electronic device. The first electronicdevice may obtain information on a calibrated geographic location of thefixed external object, based on the information on the first geographiclocation and the information on the second geographic location.

In operation 4004, the first electronic device may add the contentcorresponding to the fixed external object onto the electronic map,based on the information on the geographic location of the fixedexternal object. In other words, the first electronic device may updatethe electronic map by adding the content corresponding to the fixedexternal object onto to the electronic map.

In operation 4005, the first electronic device may transmit, to thesecond electronic device and the third electronic device, information onthe electronic map added with the content corresponding to the fixedexternal object.

Heretofore, the description has been made of an example of theelectronic device within the vehicle performing the operationsillustrated through the description of FIGS. 23 to 40, but it is onlyfor convenience of description. The operations exemplified through thedescriptions of FIGS. 23 to 40 may be performed by an electronic device(e.g., a server, etc.) located outside the vehicle.

As described above, the electronic device disposed within a movingvehicle according to various embodiments of the present disclosure mayinclude at least one camera, a sensing circuit including a transmitterconfigured to emit a signal and a receiver configured to receive areflected signal of the signal, a communication circuitry, and aprocessor operatively connected with the camera, the sensing circuit andthe communication circuitry, wherein the processor is configured to:obtain an image including a plurality of visual objects eachcorresponding to a plurality of external objects, through the camera;identify a visual object corresponding to a fixed external object of theplurality of visual objects, based on a result of analysis of theobtained image, to obtain information on the visual object; andtransmit, to a server, information on a distance between the fixedexternal object and the electronic device, and information on the visualobject, obtained through the sensing circuit, wherein the information onthe distance between the fixed external object and the electronicdevice, and the information on the visual object may be transmitted tothe server in order to add a first content corresponding to the fixedexternal object onto an electronic map.

According to an embodiment, the processor may be further configured toidentify whether the first content exists at a location on theelectronic map corresponding to a geographic location of the fixedexternal object, and obtain the image including the plurality of visualobjects, based on whether the first content exists.

According to an embodiment, the electronic device may further include adisplay.

According to an embodiment, the processor may be further configured toreceive information for displaying the first content on the electronicmap, within an application for displaying the electronic map, anddisplay the electronic map with the first content being added thereto.

According to an embodiment, the processor may be further configured toidentify whether a second content on a location on the electronic mapcorresponding to the location of the fixed external object is related tothe fixed external object, and obtain the image including the pluralityof visual objects, based on whether the second content is related to thefixed external object.

According to an embodiment, the electronic device may further include aglobal positioning system (GPS) communication circuitry.

According to an embodiment, the processor may be further configured toobtain information on the geographic location of the electronic device,through the GPS communication circuitry, identify the information on thegeographic location of the fixed external object, based on theinformation on the geographic location of the electronic device and theinformation on the distance between the fixed external object and theelectronic device, and transmit the information on the geographiclocation of the fixed external object to the server.

According to an embodiment, the information on the visual object mayinclude data on a shape of the fixed external object, data on a type ofthe fixed external object, and data on a function of the fixed externalobject.

According to an embodiment, the processor may be further configured toobtain information on a width of the fixed external object, based on thevisual object.

According to an embodiment, the data on the type of the fixed externalobject may be determined through a machine learning model, based on theinformation on the width of the fixed external object.

According to an embodiment, the processor may be further configured toobtain information on a distance between the fixed external object andthe electronic device, based on a width of the visual object and a focallength of the camera.

According to an embodiment, the processor may be further configured toclassify a plurality of visual objects into one of a dynamic object anda static object, based on a result of analysis on the obtained image.

According to an embodiment, the processor may be further configured toreceive, from an external electronic device, information on a distancebetween the fixed external object and the external electronic device,obtain information on the geographic location of the fixed externalobject, based the information on the distance between the fixed externalobject and the electronic device, and the information on the distancebetween the fixed external object and the external electronic device,and transmit the information on the geographic location of the fixedexternal object to the server.

According to an embodiment, the processor may be further configured toidentify that the external electronic device is located within apredetermined radius of the electronic device, and based on identifyingthat the external electronic device is located within the predeterminedradius of the electronic device, establish a connection with theexternal electronic device.

According to an embodiment, the processor may be further configured totransmit the obtained image to the server, and receive information on aresult of analysis of the obtained image from the server.

According to an embodiment, the obtained image may be used to extract aplurality of feature points together with a reference image, and theplurality of feature points may be points at which image brightnesschanges rapidly in the reference image or the obtained image and includean edge of a pixel, or a corner point.

According to an embodiment, the plurality of feature points may includea plurality of first feature points extracted from the reference imageand a plurality of second feature points extracted from the obtainedimage.

According to an embodiment, the processor may be further configured totransmit, to the server, information on an angle at which the obtainedimage was captured and information on a direction in which the obtainedimage was captured, together with the obtained image.

According to an embodiment, the processor may be further configured toobtain information on the width of the lane in which the moving vehicleis located, obtain information on the width of the fixed external objectbased on the information on the width of the lane, and transmit theinformation on the width of the lane of the fixed external object to theserver.

According to an embodiment, the processor may be further configured toreceive, from the fixed external object, information on the geographiclocation of the fixed external object and information on the fixedexternal object, and transmit, to the server, the information on thegeographic location of the fixed external object and the information onthe fixed external object.

According to an embodiment, the information on the geographic locationof the fixed external object and the information on the fixed externalobject may be transmitted to the server in order to add a first contentcorresponding to the fixed external object onto the electronic map.

An electronic device within a moving vehicle according to variousembodiments of the present disclosure may include at least one camera, acommunication circuitry and a processor operatively connected to thecamera and the communication circuitry, wherein the processor may beconfigured to: obtain, at a first location, a first image correspondingto the exterior of the vehicle through the camera, while the electronicdevice is moving in a first direction; identify a first visual objectcorresponding to a fixed external object of a plurality of visualobjects included in the first image; identify a first virtual planespaced apart from a center point of a lens of the camera by a focallength of the lens in the first direction, while the electronic deviceis located in the first location; identify, in the first location, afirst intersection point between a first straight line extending fromthe center point of the lens towards the fixed external object and thefirst virtual plane; after moving in the first direction from the firstlocation, obtain a second image at a second location, through thecamera, corresponding to the exterior of the vehicle; identify a secondvisual object corresponding to the fixed external object among theplurality of visual objects included in the second image; while theelectronic device is located in the second location, identify a secondvirtual plane spaced apart from the center point of the lens of thecamera by the focal length of the lens in the first direction; identify,in the second location, a second intersection point between a secondstraight line extending from the center point of the lens towards thefixed external object and the second virtual plane; determine ageographic location of the fixed external object based on the firstintersection point and the second intersection point; and transmit theinformation on the geographic location of the fixed external object tothe server.

According to an embodiment, the information on the geographic locationof the fixed external object may be transmitted to the server in orderto add the content corresponding to the fixed external object onto anelectronic map.

According to an embodiment, the electronic device may further include aglobal positioning system (GPS) communication circuitry.

According to an embodiment, the processor may be further configured toidentify the first location and the second location through the GPScommunication circuitry, determine a relative location of the fixedexternal object with respect to the second location, based on the firstintersection point and the second intersection point, and determine thegeographic location of the fixed external object based on the determinedrelative location.

According to an embodiment, the electronic device may further include aspeedometer.

According to an embodiment, the processor may be configured to identifyspeed information of the electronic device between the first locationand the second location, measured through the speedometer, identify timeinformation required for the electronic device to move from the firstlocation to the second location, obtain a distance between the firstlocation and the second location, based on the speed information and thetime information, and determine the geographic location of the fixedexternal object, based on the obtained distance between the firstlocation and the second location.

According to an embodiment, the processor may be further configured toobtain, in the first location, information on the width of the lane inwhich the vehicle is moving, identify an angle between the firstdirection and the first straight line, and determine the geographiclocation of the fixed external object based on the width of the lane andthe angle.

According to an embodiment, the processor may be further configured toobtain data on a type of the fixed external object, based on the firstvisual object and the second visual object, and transmit the data on thetype of the fixed external object to the server.

An electronic device within a vehicle moving in a first directionaccording to various embodiments of the present disclosure may include afirst camera, a second camera spaced apart from the first camera by afirst distance, a communication circuitry, a processor operativelyconnected to the first camera, the second camera and the communicationcircuitry, wherein the processor may be configured to: obtain an imagecorresponding to the exterior of the vehicle through at least one of thefirst camera and the second camera; identify a visual objectcorresponding to a fixed external object of a plurality of visualobjects included in the image; identify a first virtual plane spacedapart from a center point of a first lens of the first camera by a focuslength of the first lens in the first direction; identify a secondvirtual plane spaced apart from a center point of a second lens of thesecond camera by a focal length of the second lens in the firstdirection; identify a first intersection point between a first straightline extending from the center point of the first lens towards the fixedexternal object and the first virtual plane; identify a secondintersection point between a second straight line extending from thecenter point of the second lens towards the fixed external object andthe second virtual plane; determine the geographic location of the fixedexternal object based on the first intersection point and the secondintersection point; and transmit the information on the geographiclocation of the fixed external object to the server.

According to an embodiment, the information on the geographic locationof the fixed external object may be transmitted to the server in orderto add the content corresponding to the fixed external object onto anelectronic map.

According to various embodiments, a first electronic device may includea memory and a processor operatively connected to the memory, whereinthe processor may be configured to: obtain, from a second electronicdevice, a first image including a first visual object corresponding to afixed external object; obtain, from a third electronic device, a secondimage including a second visual object corresponding to the fixedexternal object; obtain information on the geographic location of thefixed external object, based on the first image and the second image;add a content corresponding to the fixed external object onto anelectronic map, based on the information on the geographic location ofthe fixed external object; and transmit the information on theelectronic map with the content being added thereto, to the secondelectronic device and the third electronic device.

Meanwhile, the control methods according to various exemplaryembodiments of the present disclosure set forth heretofore may beimplemented in a program and provided to a server or any other similardevices. Accordingly, the respective devices implementing theabove-described control methods may access the server or other similardevices, in which the program is stored, for downloading the same.

Further, the control method according to various exemplary embodimentsof the present disclosure described above may be implemented in aprogram and stored and provided in various non-transitorycomputer-readable mediums. The non-transitory readable medium is not amedium to store data for a short period of time, such as e.g., aregister, a cache, a memory, or the like, and it means a medium tosemi-permanently stores data and is readable by a computing device.Specifically, the various applications or programs described above maybe stored and provided in the non-transitory computer-readable mediumsuch as e.g., a compact disk (CD), a digital versatile disk (DVD), ahard disk, a Blu-ray disk, a universal serial bus (USB), a memory card,a read only memory (ROM), or the like.

Although exemplary embodiments of the present disclosure have beenillustrated and described hereinabove, the present invention is notlimited to the above-mentioned specific exemplary embodiments, but maybe variously modified by those skilled in the art to which the presentinvention pertains without departing from the scope and spirit of thepresent invention as disclosed in the accompanying claims. Thesemodifications should also be understood to fall within the scope of thepresent invention.

What is claimed is:
 1. An electronic device disposed within a movingvehicle, comprising: at least one camera; a sensing circuit including atransmitter configured to emit a signal and a receiver configured toreceive a reflected signal of the signal; a communication circuitry; anda processor operatively connected with the camera, the sensing circuitand the communication circuitry, wherein the processor is configured to:obtain an image including a plurality of visual objects eachcorresponding to a plurality of external objects, through the camera;identify a visual object corresponding to a fixed external object of theplurality of visual objects, based on a result of analysis of theobtained image, to obtain information on the visual object; andtransmit, to a server, information on a distance between the fixedexternal object and the electronic device, and information on the visualobject, obtained through the sensing circuit; wherein the information onthe distance between the fixed external object and the electronicdevice, and the information on the visual object are transmitted to theserver in order to add a first content corresponding to the fixedexternal object onto an electronic map.
 2. The electronic deviceaccording to claim 1, wherein the processor is further configured to:identify whether the first content exists at a location on theelectronic map corresponding to a geographic location of the fixedexternal object; and obtain the image including the plurality of visualobjects, based on whether the first content exists.
 3. The electronicdevice according to claim 2, wherein: the electronic device furtherincludes a display, and the processor is further configured to: receiveinformation for displaying the first content on the electronic map,within an application for displaying the electronic map; and display theelectronic map with the first content being added thereto.
 4. Theelectronic device according to claim 1, wherein the processor is furtherconfigured to: identify whether a second content on a location on theelectronic map corresponding to the location of the fixed externalobject is related to the fixed external object; and obtain the imageincluding the plurality of visual objects, based on whether the secondcontent is related to the fixed external object.
 5. The electronicdevice according to claim 1, wherein the electronic device furtherincludes a global positioning system (GPS) communication circuitry, andthe processor is further configured to: obtain information on ageographic location of the electronic device, through the GPScommunication circuitry; identify information on a geographic locationof the fixed external object, based on the information on the geographiclocation of the electronic device and the information on the distancebetween the fixed external object and the electronic device; andtransmit the information on the geographic location of the fixedexternal object to the server.
 6. The electronic device according toclaim 1, wherein the information on the visual object includes data on ashape of the fixed external object, data on a type of the fixed externalobject, and data on a function of the fixed external object.
 7. Theelectronic device according to claim 6, wherein the processor is furtherconfigured to obtain information on a width of the fixed externalobject, based on the visual object.
 8. The electronic device accordingto claim 7, wherein the data on the type of the fixed external object isdetermined through a machine learning model, based on the information onthe width of the fixed external object.
 9. The electronic deviceaccording to claim 7, wherein the processor is further configured toobtain the information on the distance between the fixed external objectand the electronic device, based on a width of the visual object and afocal length of the camera.
 10. The electronic device according to claim1, wherein the processor is further configured to classify the pluralityof visual objects into one of a dynamic object and a static object,based on a result of analysis on the obtained image.
 11. The electronicdevice according to claim 1, wherein the processor is further configuredto: receive, from an external electronic device, the information on thedistance between the fixed external object and the external electronicdevice; obtain information on a geographic location of the fixedexternal object, based the information on the distance between the fixedexternal object and the electronic device, and the information on thedistance between the fixed external object and the external electronicdevice; and transmit the information on the geographic location of thefixed external object to the server.
 12. The electronic device accordingto claim 11, wherein the processor is further configured to: identifythat the external electronic device is located within a predeterminedradius of the electronic device; and based on identifying that theexternal electronic device is located within the predetermined radius ofthe electronic device, establish a connection with the externalelectronic device.
 13. The electronic device according to claim 1,wherein the processor is further configured to: transmit the obtainedimage to the server; and receive information on the result of analysisof the obtained image from the server.
 14. The electronic deviceaccording to claim 13, wherein: the obtained image is used to extract aplurality of feature points together with a reference image; and theplurality of feature points are points at which image brightness changesrapidly in the reference image or the obtained image and include an edgeof a pixel, or a corner point.
 15. The electronic device according toclaim 14, wherein the plurality of feature points include a plurality offirst feature points extracted from the reference image and a pluralityof second feature points extracted from the obtained image.
 16. Theelectronic device according to claim 13, wherein the processor isfurther configured to transmit, to the server, information on an angleat which the obtained image was captured and information on a directionin which the obtained image was captured, together with the obtainedimage.
 17. The electronic device according to claim 1, wherein theprocessor is further configured to: obtain information on a width of thelane in which the moving vehicle is located; obtain information on thewidth of the fixed external object, based on the information on thewidth of the lane; and transmit the information on the width of the laneof the fixed external object to the server.
 18. The electronic deviceaccording to claim 1, wherein the processor is further configured to:receive, from the fixed external object, information on a geographiclocation of the fixed external object and information on the fixedexternal object; and transmit, to the server, the information on thegeographic location of the fixed external object and the information onthe fixed external object; and the information on the geographiclocation of the fixed external object and the information on the fixedexternal object are transmitted to the server in order to add a firstcontent corresponding to the fixed external object onto the electronicmap.
 19. An electronic device disposed within a moving vehicle,comprising: at least one camera; a communication circuitry; and aprocessor operatively connected to the camera and the communicationcircuitry, wherein the processor is configured to: obtain, at a firstlocation, a first image corresponding to the exterior of the vehiclethrough the camera, while the electronic device is moving in a firstdirection; identify a first visual object corresponding to a fixedexternal object of a plurality of visual objects included in the firstimage; identify a first virtual plane spaced apart from a center pointof a lens of the camera by a focal length of the lens in the firstdirection, while the electronic device is located in the first location;identify, at the first location, a first intersection point between afirst straight line extending from the center point of the lens towardsthe fixed external object and the first virtual plane; after moving inthe first direction from the first location, obtain a second image at asecond location, through the camera, corresponding to the exterior ofthe vehicle; identify a second visual object corresponding to the fixedexternal object among the plurality of visual objects included in thesecond image; while the electronic device is located in the secondlocation, identify a second virtual plane spaced apart from the centerpoint of the lens of the camera by the focal length of the lens in thefirst direction; identify, at the second location, a second intersectionpoint between a second straight line extending from the center point ofthe lens towards the fixed external object and the second virtual plane;determine a geographic location of the fixed external object, based onthe first intersection point and the second intersection point; andtransmit the information on the geographic location of the fixedexternal object to the server; wherein the information on the geographiclocation of the fixed external object is transmitted to the server inorder to add a content corresponding to the fixed external object ontoan electronic map.
 20. A first electronic device comprising: a memory;and a processor operatively connected to the memory, wherein theprocessor is configured to: obtain, from a second electronic device, afirst image including a first visual object corresponding to a fixedexternal object; obtain, from a third electronic device, a second imageincluding a second visual object corresponding to the fixed externalobject; obtain information on a geographic location of the fixedexternal object, based on the first image and the second image; add acontent corresponding to the fixed external object onto an electronicmap, based on the information on the geographic location of the fixedexternal object; and transmit the information on the electronic map withthe content being added thereto, to the second electronic device and thethird electronic device.